Video or image coding based on mapped luma component and scaled chroma component

ABSTRACT

According to the disclosure of the present document, a linear reshaper is used for LMCS and a single chroma residual scaling factor, which is directly signaled during chroma scaling of the LMCS, can be used. In addition, the number of flexible bins can be used for luma mapping so that the resource/cost (of software or hardware) required for an LMCS procedure is minimized, and the latency of coding is removed so that the LMCS procedure can be efficiently performed.

BACKGROUND OF THE DISCLOSURE Field of the Disclosure

The present document relates to video or image coding based on mappedluma component and scaled chroma component.

Related Art

Recently, demand for high-resolution, high-quality image/video such as4K or 8K or higher ultra high definition (UHD) image/video has increasedin various fields. As image/video data has high resolution and highquality, the amount of information or bits to be transmitted increasesrelative to the existing image/video data, and thus, transmitting imagedata using a medium such as an existing wired/wireless broadband line oran existing storage medium or storing image/video data using existingstorage medium increase transmission cost and storage cost.

In addition, interest and demand for immersive media such as virtualreality (VR) and artificial reality (AR) content or holograms hasrecently increased and broadcasting for image/video is havingcharacteristics different from reality images such as game images hasincreased.

Accordingly, a highly efficient image/video compression technology isrequired to effectively compress, transmit, store, and reproduceinformation of a high-resolution, high-quality image/video havingvarious characteristics as described above.

In addition, a luma mapping with chroma scaling (LMCS) procedure isperformed in order to improve compression efficiency and increasesubjective/objective visual quality, and there is a discussion about amethod for efficiently signaling information about the LMCS procedure.

SUMMARY

According to an embodiment of the present document, a method and anapparatus for increasing image/video coding efficiency are provided.

According to an embodiment of the present document, an efficientfiltering application method and apparatus are provided.

According to an embodiment of the present document, an efficient LMCSapplication method and apparatus are provided.

According to an embodiment of the present document, the LMCS codewords(or a range thereof) may be limited.

According to an embodiment of this document, a single chroma residualscaling factor directly signaled in chroma scaling of LMCS may be used.

According to an embodiment of this document, linear mapping (linearLMCS) may be used.

According to an embodiment of the present document, information aboutpivot points required for linear mapping may be explicitly signaled.

According to an embodiment of this document, a flexible number of binsmay be used for luma mapping.

According to an embodiment of the present document, a video/imagedecoding method performed by a decoding apparatus is provided.

According to an embodiment of the present document, a decoding apparatusfor performing video/image decoding is provided.

According to an embodiment of the present document, a video/imageencoding method performed by an encoding apparatus is provided.

According to an embodiment of the present document, an encodingapparatus for performing video/image encoding is provided.

According to one embodiment of the present document, there is provided acomputer-readable digital storage medium in which encoded video/imageinformation, generated according to the video/image encoding methoddisclosed in at least one of the embodiments of the present document, isstored.

According to an embodiment of the present document, there is provided acomputer-readable digital storage medium in which encoded information orencoded video/image information, causing to perform the video/imagedecoding method disclosed in at least one of the embodiments of thepresent document by the decoding apparatus, is stored.

Advantageous Effects

According to an embodiment of the present document, overall image/videocompression efficiency may be improved.

According to an embodiment of the present document, subjective/objectivevisual quality may be improved through efficient filtering.

According to an embodiment of the present document, the LMCS process forimage/video coding may be efficiently performed.

According to an embodiment of the present document, it is possible tominimize resources/costs (of software or hardware) required for the LMCSprocess.

According to an embodiment of the present document, hardwareimplementation for the LMCS process may be facilitated.

According to an embodiment of the present document, a division operationrequired for derivation of LMCS codewords in mapping (reshaping) can beremoved or minimized by constraint of the LMCS codewords (or rangethereof).

According to an embodiment of the present document, latency according topiecewise index identification may be removed by using a single chromaresidual scaling factor.

According to an embodiment of the present document, a chroma residualscaling process can be performed without depending on (reconstructionof) a luma block by using the linear mapping in LMCS, and thus latencyin scaling can be removed.

According to an embodiment of this document, mapping efficiency in LMCSmay be increased.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a video/image coding system to whichthe embodiments of the present document may be applied.

FIG. 2 is a diagram schematically illustrating a configuration of avideo/image encoding apparatus to which the embodiments of the presentdocument may be applied.

FIG. 3 is a diagram schematically illustrating a configuration of avideo/image decoding apparatus to which the embodiments of the presentdocument may be applied.

FIG. 4 exemplarily shows a hierarchical structure for a codedimage/video.

FIG. 5 exemplarily illustrates a hierarchical structure of a CVSaccording to an embodiment of the present document.

FIG. 6 illustrates an exemplary LMCS structure according to anembodiment of the present document.

FIG. 7 illustrates an LMCS structure according to another embodiment ofthe present document.

FIG. 8 shows a graph representing an exemplary forward mapping.

FIG. 9 is a flowchart illustrating a method for deriving a chromaresidual scaling index according to an embodiment of the presentdocument.

FIG. 10 illustrates a linear fitting of pivot points according to anembodiment of the present document.

FIG. 11 illustrates one example of linear reshaping (or linearreshaping, linear mapping) according to an embodiment of the presentdocument.

FIG. 12 and FIG. 13 schematically show an example of a video/imageencoding method and related components according to embodiment(s) of thepresent document.

FIG. 14 and FIG. 15 schematically show an example of an image/videodecoding method and related components according to an embodiment of thepresent document.

FIG. 16 shows an example of a content streaming system to whichembodiments disclosed in the present document may be applied.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

The present document may be modified in various forms, and specificembodiments thereof will be described and shown in the drawings.However, the embodiments are not intended for limiting the presentdocument. The terms used in the following description are used to merelydescribe specific embodiments, but are not intended to limit the presentdocument. An expression of a singular number includes an expression ofthe plural number, so long as it is clearly read differently. The termssuch as “include” and “have” are intended to indicate that features,numbers, steps, operations, elements, components, or combinationsthereof used in the following description exist and it should be thusunderstood that the possibility of existence or addition of one or moredifferent features, numbers, steps, operations, elements, components, orcombinations thereof is not excluded.

Meanwhile, each configuration in the drawings described in the presentdocument is shown independently for the convenience of descriptionregarding different characteristic functions, and does not mean thateach configuration is implemented as separate hardware or separatesoftware. For example, two or more components among each component maybe combined to form one component, or one component may be divided intoa plurality of components. Embodiments in which each component isintegrated and/or separated are also included in the scope of thedisclosure of the present document.

Hereinafter, examples of the present embodiment will be described indetail with reference to the accompanying drawings. In addition, likereference numerals are used to indicate like elements throughout thedrawings, and the same descriptions on the like elements will beomitted.

FIG. 1 illustrates an example of a video/image coding system to whichthe embodiments of the present document may be applied.

Referring to FIG. 1, a video/image coding system may include a firstdevice (a source device) and a second device (a reception device). Thesource device may transmit encoded video/image information or data tothe reception device through a digital storage medium or network in theform of a file or streaming.

The source device may include a video source, an encoding apparatus, anda transmitter. The receiving device may include a receiver, a decodingapparatus, and a renderer. The encoding apparatus may be called avideo/image encoding apparatus, and the decoding apparatus may be calleda video/image decoding apparatus. The transmitter may be included in theencoding apparatus. The receiver may be included in the decodingapparatus. The renderer may include a display, and the display may beconfigured as a separate device or an external component.

The video source may acquire video/image through a process of capturing,synthesizing, or generating the video/image. The video source mayinclude a video/image capture device and/or a video/image generatingdevice. The video/image capture device may include, for example, one ormore cameras, video/image archives including previously capturedvideo/images, and the like. The video/image generating device mayinclude, for example, computers, tablets and smartphones, and may(electronically) generate video/images. For example, a virtualvideo/image may be generated through a computer or the like. In thiscase, the video/image capturing process may be replaced by a process ofgenerating related data.

The encoding apparatus may encode input video/image. The encodingapparatus may perform a series of procedures such as prediction,transform, and quantization for compaction and coding efficiency. Theencoded data (encoded video/image information) may be output in the formof a bitstream.

The transmitter may transmit the encoded image/image information or dataoutput in the form of a bitstream to the receiver of the receivingdevice through a digital storage medium or a network in the form of afile or streaming. The digital storage medium may include variousstorage mediums such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, and thelike. The transmitter may include an element for generating a media filethrough a predetermined file format and may include an element fortransmission through a broadcast/communication network. The receiver mayreceive/extract the bitstream and transmit the received bitstream to thedecoding apparatus.

The decoding apparatus may decode the video/image by performing a seriesof procedures such as dequantization, inverse transform, and predictioncorresponding to the operation of the encoding apparatus.

The renderer may render the decoded video/image. The renderedvideo/image may be displayed through the display.

The present document relates to video/image coding. For example, amethod/embodiment disclosed in the present document may be applied to amethod disclosed in the versatile video coding (VVC) standard, theessential video coding (EVC) standard, the AOMedia Video 1 (AV1)standard, the 2nd generation of audio video coding standard (AVS2) orthe next generation video/image coding standard (e.g., H.267, H.268, orthe like).

The present document suggests various embodiments of video/image coding,and the above embodiments may also be performed in combination with eachother unless otherwise specified.

In the present document, a video may refer to a series of images overtime. A picture generally refers to the unit representing one image at aparticular time frame, and a slice/tile refers to the unit constitutinga part of the picture in terms of coding. A slice/tile may include oneor more coding tree units (CTUs). One picture may consist of one or moreslices/tiles. One picture may consist of one or more tile groups. Onetile group may include one or more tiles. A brick may represent arectangular region of CTU rows within a tile in a picture. A tile may bepartitioned into a multiple bricks, each of which may be constructedwith one or more CTU rows within the tile. A tile that is notpartitioned into multiple bricks may also be referred to as a brick. Abrick scan may represent a specific sequential ordering of CTUspartitioning a picture, wherein the CTUs may be ordered in a CTU rasterscan within a brick, and bricks within a tile may be orderedconsecutively in a raster scan of the bricks of the tile, and tiles in apicture may be ordered consecutively in a raster scan of the tiles ofthe picture. A tile is a rectangular region of CTUs within a particulartile column and a particular tile row in a picture. The tile column is arectangular region of CTUs having a height equal to the height of thepicture and a width specified by syntax elements in the pictureparameter set. The tile row is a rectangular region of CTUs having aheight specified by syntax elements in the picture parameter set and awidth equal to the width of the picture. A tile scan is a specificsequential ordering of CTUs partitioning a picture in which the CTUs areordered consecutively in CTU raster scan in a tile whereas tiles in apicture are ordered consecutively in a raster scan of the tiles of thepicture. A slice includes an integer number of bricks of a picture thatmay be exclusively contained in a single NAL unit. A slice may consistsof either a number of complete tiles or only a consecutive sequence ofcomplete bricks of one tile. In the present document, a tile group and aslice may be used in place of each other. For example, in the presentdocument, a tile group/tile group header may be referred to as aslice/slice header.

Meanwhile, one picture may be divided into two or more subpictures. Asubpicture may be a rectangular region of one or more slices within apicture.

A pixel or a pel may mean a smallest unit constituting one picture (orimage). Also, ‘sample’ may be used as a term corresponding to a pixel. Asample may generally represent a pixel or a value of a pixel, and mayrepresent only a pixel/pixel value of a luma component or only apixel/pixel value of a chroma component.

A unit may represent a basic unit of image processing. The unit mayinclude at least one of a specific region of the picture and informationrelated to the region. One unit may include one luma block and twochroma (ex. cb, cr) blocks. The unit may be used interchangeably withterms such as block or area in some cases. In a general case, an M×Nblock may include samples (or sample arrays) or a set (or array) oftransform coefficients of M columns and N rows. Alternatively, thesample may mean a pixel value in the spatial domain, and when such apixel value is transformed to the frequency domain, it may mean atransform coefficient in the frequency domain.

In the present document, “A or B” may mean “only A”, “only B” or “both Aand B”. In other words, “A or B” in the present document may beinterpreted as “A and/or B”. For example, in the present document “A, Bor C (A, B or C)” means “only A”, “only B”, “only C”, or “anycombination of A, B and C”.

A slash (/) or comma (comma) used in the present document may mean“and/or”. For example, “A/B” may mean “A and/or B”. Accordingly, “A/B”may mean “only A”, “only B”, or “both A and B”. For example, “A, B, C”may mean “A, B, or C”.

In the present document, “at least one of A and B” may mean “only A”,“only B” or “both A and B”. Also, in the present document, theexpression “at least one of A or B” or “at least one of A and/or B” maybe interpreted the same as “at least one of A and B”.

Also, in the present document, “at least one of A, B and C” means “onlyA”, “only B”, “only C”, or “any combination of A, B and C”. Also, “atleast one of A, B or C” or “at least one of A, B and/or C” may mean “atleast one of A, B and C”.

Also, parentheses used in the present document may mean “for example”.Specifically, when “prediction (intra prediction)” is indicated, “intraprediction” may be proposed as an example of “prediction”. In otherwords, “prediction” in the present document is not limited to “intraprediction”, and “intra prediction” may be proposed as an example of“prediction”. Also, even when “prediction (i.e., intra prediction)” isindicated, “intra prediction” may be proposed as an example of“prediction”.

Technical features that are individually described in one drawing in thepresent document may be implemented individually or simultaneously.

FIG. 2 is a diagram schematically illustrating a configuration of avideo/image encoding apparatus to which the embodiments of the presentdocument may be applied. Hereinafter, what is referred to as the videoencoding apparatus may include an image encoding apparatus.

Referring to FIG. 2, the encoding apparatus 200 includes an imagepartitioner 210, a predictor 220, a residual processor 230, and anentropy encoder 240, an adder 250, a filter 260, and a memory 270. Thepredictor 220 may include an inter predictor 221 and an intra predictor222. The residual processor 230 may include a transformer 232, aquantizer 233, a dequantizer 234, and an inverse transformer 235. Theresidual processor 230 may further include a subtractor 231. The adder250 may be called a reconstructor or a reconstructed block generator.The image partitioner 210, the predictor 220, the residual processor230, the entropy encoder 240, the adder 250, and the filter 260 may beconfigured by at least one hardware component (ex. An encoder chipset orprocessor) according to an embodiment. In addition, the memory 270 mayinclude a decoded picture buffer (DPB) or may be configured by a digitalstorage medium. The hardware component may further include the memory270 as an internal/external component.

The image partitioner 210 may partition an input image (or a picture ora frame) input to the encoding apparatus 200 into one or moreprocessors. For example, the processor may be called a coding unit (CU).In this case, the coding unit may be recursively partitioned accordingto a quad-tree binary-tree ternary-tree (QTBTTT) structure from a codingtree unit (CTU) or a largest coding unit (LCU). For example, one codingunit may be partitioned into a plurality of coding units of a deeperdepth based on a quad tree structure, a binary tree structure, and/or aternary structure. In this case, for example, the quad tree structuremay be applied first and the binary tree structure and/or ternarystructure may be applied later. Alternatively, the binary tree structuremay be applied first. The coding procedure according to the presentdisclosure may be performed based on the final coding unit that is nolonger partitioned. In this case, the largest coding unit may be used asthe final coding unit based on coding efficiency according to imagecharacteristics, or if necessary, the coding unit may be recursivelypartitioned into coding units of deeper depth and a coding unit havingan optimal size may be used as the final coding unit. Here, the codingprocedure may include a procedure of prediction, transform, andreconstruction, which will be described later. As another example, theprocessor may further include a prediction unit (PU) or a transform unit(TU). In this case, the prediction unit and the transform unit may besplit or partitioned from the aforementioned final coding unit. Theprediction unit may be a unit of sample prediction, and the transformunit may be a unit for deriving a transform coefficient and/or a unitfor deriving a residual signal from the transform coefficient.

The unit may be used interchangeably with terms such as block or area insome cases. In a general case, an M×N block may represent a set ofsamples or transform coefficients composed of M columns and N rows. Asample may generally represent a pixel or a value of a pixel, mayrepresent only a pixel/pixel value of a luma component or represent onlya pixel/pixel value of a chroma component. A sample may be used as aterm corresponding to one picture (or image) for a pixel or a pel.

In the encoding apparatus 200, a prediction signal (predicted block,prediction sample array) output from the inter predictor 221 or theintra predictor 222 is subtracted from an input image signal (originalblock, original sample array) to generate a residual signal residualblock, residual sample array), and the generated residual signal istransmitted to the transformer 232. In this case, as shown, a unit forsubtracting a prediction signal (predicted block, prediction samplearray) from the input image signal (original block, original samplearray) in the encoder 200 may be called a subtractor 231. The predictormay perform prediction on a block to be processed (hereinafter, referredto as a current block) and generate a predicted block includingprediction samples for the current block. The predictor may determinewhether intra prediction or inter prediction is applied on a currentblock or CU basis. As described later in the description of eachprediction mode, the predictor may generate various information relatedto prediction, such as prediction mode information, and transmit thegenerated information to the entropy encoder 240. The information on theprediction may be encoded in the entropy encoder 240 and output in theform of a bitstream.

The intra predictor 222 may predict the current block by referring tothe samples in the current picture. The referred samples may be locatedin the neighborhood of the current block or may be located apartaccording to the prediction mode. In the intra prediction, predictionmodes may include a plurality of non-directional modes and a pluralityof directional modes. The non-directional mode may include, for example,a DC mode and a planar mode. The directional mode may include, forexample, 33 directional prediction modes or 65 directional predictionmodes according to the degree of detail of the prediction direction.However, this is merely an example, more or less directional predictionmodes may be used depending on a setting. The intra predictor 222 maydetermine the prediction mode applied to the current block by using aprediction mode applied to a neighboring block.

The inter predictor 221 may derive a predicted block for the currentblock based on a reference block (reference sample array) specified by amotion vector on a reference picture. Here, in order to reduce theamount of motion information transmitted in the inter prediction mode,the motion information may be predicted in units of blocks, sub-blocks,or samples based on correlation of motion information between theneighboring block and the current block. The motion information mayinclude a motion vector and a reference picture index. The motioninformation may further include inter prediction direction (L0prediction, L1 prediction, Bi prediction, etc.) information. In the caseof inter prediction, the neighboring block may include a spatialneighboring block present in the current picture and a temporalneighboring block present in the reference picture. The referencepicture including the reference block and the reference pictureincluding the temporal neighboring block may be the same or different.The temporal neighboring block may be called a collocated referenceblock, a co-located CU (colCU), and the like, and the reference pictureincluding the temporal neighboring block may be called a collocatedpicture (colPic). For example, the inter predictor 221 may configure amotion information candidate list based on neighboring blocks andgenerate information indicating which candidate is used to derive amotion vector and/or a reference picture index of the current block.Inter prediction may be performed based on various prediction modes. Forexample, in the case of a skip mode and a merge mode, the interpredictor 221 may use motion information of the neighboring block asmotion information of the current block. In the skip mode, unlike themerge mode, the residual signal may not be transmitted. In the case ofthe motion vector prediction (MVP) mode, the motion vector of theneighboring block may be used as a motion vector predictor and themotion vector of the current block may be indicated by signaling amotion vector difference.

The predictor 220 may generate a prediction signal based on variousprediction methods described below. For example, the predictor may notonly apply intra prediction or inter prediction to predict one block butalso simultaneously apply both intra prediction and inter prediction.This may be called combined inter and intra prediction (CIIP). Inaddition, the predictor may be based on an intra block copy (IBC)prediction mode or a palette mode for prediction of a block. The IBCprediction mode or palette mode may be used for content image/videocoding of a game or the like, for example, screen content coding (SCC).The IBC basically performs prediction in the current picture but may beperformed similarly to inter prediction in that a reference block isderived in the current picture. That is, the IBC may use at least one ofthe inter prediction techniques described in the present disclosure. Thepalette mode may be considered as an example of intra coding or intraprediction. When the palette mode is applied, a sample value within apicture may be signaled based on information on the palette table andthe palette index.

The prediction signal generated by the predictor (including the interpredictor 221 and/or the intra predictor 222) may be used to generate areconstructed signal or to generate a residual signal. The transformer232 may generate transform coefficients by applying a transformtechnique to the residual signal. For example, the transform techniquemay include at least one of a discrete cosine transform (DCT), adiscrete sine transform (DST), a karhunen-loeve transform (KLT), agraph-based transform (GBT), or a conditionally non-linear transform(CNT). Here, the GBT means transform obtained from a graph whenrelationship information between pixels is represented by the graph. TheCNT refers to transform generated based on a prediction signal generatedusing all previously reconstructed pixels. In addition, the transformprocess may be applied to square pixel blocks having the same size ormay be applied to blocks having a variable size rather than square.

The quantizer 233 may quantize the transform coefficients and transmitthem to the entropy encoder 240 and the entropy encoder 240 may encodethe quantized signal (information on the quantized transformcoefficients) and output a bitstream. The information on the quantizedtransform coefficients may be referred to as residual information. Thequantizer 233 may rearrange block type quantized transform coefficientsinto a one-dimensional vector form based on a coefficient scanning orderand generate information on the quantized transform coefficients basedon the quantized transform coefficients in the one-dimensional vectorform. Information on transform coefficients may be generated. Theentropy encoder 240 may perform various encoding methods such as, forexample, exponential Golomb, context-adaptive variable length coding(CAVLC), context-adaptive binary arithmetic coding (CABAC), and thelike. The entropy encoder 240 may encode information necessary forvideo/image reconstruction other than quantized transform coefficients(ex. values of syntax elements, etc.) together or separately. Encodedinformation (ex. encoded video/image information) may be transmitted orstored in units of NALs (network abstraction layer) in the form of abitstream. The video/image information may further include informationon various parameter sets such as an adaptation parameter set (APS), apicture parameter set (PPS), a sequence parameter set (SPS), or a videoparameter set (VPS). In addition, the video/image information mayfurther include general constraint information. In the presentdisclosure, information and/or syntax elements transmitted/signaled fromthe encoding apparatus to the decoding apparatus may be included invideo/picture information. The video/image information may be encodedthrough the above-described encoding procedure and included in thebitstream. The bitstream may be transmitted over a network or may bestored in a digital storage medium. The network may include abroadcasting network and/or a communication network, and the digitalstorage medium may include various storage media such as USB, SD, CD,DVD, Blu-ray, HDD, SSD, and the like. A transmitter (not shown)transmitting a signal output from the entropy encoder 240 and/or astorage unit (not shown) storing the signal may be included asinternal/external element of the encoding apparatus 200, andalternatively, the transmitter may be included in the entropy encoder240.

The quantized transform coefficients output from the quantizer 233 maybe used to generate a prediction signal. For example, the residualsignal (residual block or residual samples) may be reconstructed byapplying dequantization and inverse transform to the quantized transformcoefficients through the dequantizer 234 and the inverse transformer235. The adder 250 adds the reconstructed residual signal to theprediction signal output from the inter predictor 221 or the intrapredictor 222 to generate a reconstructed signal (reconstructed picture,reconstructed block, reconstructed sample array). If there is noresidual for the block to be processed, such as a case where the skipmode is applied, the predicted block may be used as the reconstructedblock. The adder 250 may be called a reconstructor or a reconstructedblock generator. The generated reconstructed signal may be used forintra prediction of a next block to be processed in the current pictureand may be used for inter prediction of a next picture through filteringas described below.

Meanwhile, luma mapping with chroma scaling (LMCS) may be applied duringpicture encoding and/or reconstruction.

The filter 260 may improve subjective/objective image quality byapplying filtering to the reconstructed signal. For example, the filter260 may generate a modified reconstructed picture by applying variousfiltering methods to the reconstructed picture and store the modifiedreconstructed picture in the memory 270, specifically, a DPB of thememory 270. The various filtering methods may include, for example,deblocking filtering, a sample adaptive offset, an adaptive loop filter,a bilateral filter, and the like. The filter 260 may generate variousinformation related to the filtering and transmit the generatedinformation to the entropy encoder 240 as described later in thedescription of each filtering method. The information related to thefiltering may be encoded by the entropy encoder 240 and output in theform of a bitstream.

The modified reconstructed picture transmitted to the memory 270 may beused as the reference picture in the inter predictor 221. When the interprediction is applied through the encoding apparatus, predictionmismatch between the encoding apparatus 200 and the decoding apparatus300 may be avoided and encoding efficiency may be improved.

The DPB of the memory 270 DPB may store the modified reconstructedpicture for use as a reference picture in the inter predictor 221. Thememory 270 may store the motion information of the block from which themotion information in the current picture is derived (or encoded) and/orthe motion information of the blocks in the picture that have alreadybeen reconstructed. The stored motion information may be transmitted tothe inter predictor 221 and used as the motion information of thespatial neighboring block or the motion information of the temporalneighboring block. The memory 270 may store reconstructed samples ofreconstructed blocks in the current picture and may transfer thereconstructed samples to the intra predictor 222.

FIG. 3 is a schematic diagram illustrating a configuration of avideo/image decoding apparatus to which the embodiment(s) of the presentdisclosure may be applied.

Referring to FIG. 3, the decoding apparatus 300 may include an entropydecoder 310, a residual processor 320, a predictor 330, an adder 340, afilter 350, a memory 360. The predictor 330 may include an interpredictor 331 and an intra predictor 332. The residual processor 320 mayinclude a dequantizer 321 and an inverse transformer 321. The entropydecoder 310, the residual processor 320, the predictor 330, the adder340, and the filter 350 may be configured by a hardware component (ex. Adecoder chipset or a processor) according to an embodiment. In addition,the memory 360 may include a decoded picture buffer (DPB) or may beconfigured by a digital storage medium. The hardware component mayfurther include the memory 360 as an internal/external component.

When a bitstream including video/image information is input, thedecoding apparatus 300 may reconstruct an image corresponding to aprocess in which the video/image information is processed in theencoding apparatus of FIG. 2. For example, the decoding apparatus 300may derive units/blocks based on block partition related informationobtained from the bitstream. The decoding apparatus 300 may performdecoding using a processor applied in the encoding apparatus. Thus, theprocessor of decoding may be a coding unit, for example, and the codingunit may be partitioned according to a quad tree structure, binary treestructure and/or ternary tree structure from the coding tree unit or thelargest coding unit. One or more transform units may be derived from thecoding unit. The reconstructed image signal decoded and output throughthe decoding apparatus 300 may be reproduced through a reproducingapparatus.

The decoding apparatus 300 may receive a signal output from the encodingapparatus of FIG. 2 in the form of a bitstream, and the received signalmay be decoded through the entropy decoder 310. For example, the entropydecoder 310 may parse the bitstream to derive information (ex.video/image information) necessary for image reconstruction (or picturereconstruction). The video/image information may further includeinformation on various parameter sets such as an adaptation parameterset (APS), a picture parameter set (PPS), a sequence parameter set(SPS), or a video parameter set (VPS). In addition, the video/imageinformation may further include general constraint information. Thedecoding apparatus may further decode picture based on the informationon the parameter set and/or the general constraint information.Signaled/received information and/or syntax elements described later inthe present disclosure may be decoded may decode the decoding procedureand obtained from the bitstream. For example, the entropy decoder 310decodes the information in the bitstream based on a coding method suchas exponential Golomb coding, CAVLC, or CABAC, and output syntaxelements required for image reconstruction and quantized values oftransform coefficients for residual. More specifically, the CABACentropy decoding method may receive a bin corresponding to each syntaxelement in the bitstream, determine a context model using a decodingtarget syntax element information, decoding information of a decodingtarget block or information of a symbol/bin decoded in a previous stage,and perform an arithmetic decoding on the bin by predicting aprobability of occurrence of a bin according to the determined contextmodel, and generate a symbol corresponding to the value of each syntaxelement. In this case, the CABAC entropy decoding method may update thecontext model by using the information of the decoded symbol/bin for acontext model of a next symbol/bin after determining the context model.The information related to the prediction among the information decodedby the entropy decoder 310 may be provided to the predictor (the interpredictor 332 and the intra predictor 331), and the residual value onwhich the entropy decoding was performed in the entropy decoder 310,that is, the quantized transform coefficients and related parameterinformation, may be input to the residual processor 320. The residualprocessor 320 may derive the residual signal (the residual block, theresidual samples, the residual sample array). In addition, informationon filtering among information decoded by the entropy decoder 310 may beprovided to the filter 350. Meanwhile, a receiver (not shown) forreceiving a signal output from the encoding apparatus may be furtherconfigured as an internal/external element of the decoding apparatus300, or the receiver may be a component of the entropy decoder 310.Meanwhile, the decoding apparatus according to the present disclosuremay be referred to as a video/image/picture decoding apparatus, and thedecoding apparatus may be classified into an information decoder(video/image/picture information decoder) and a sample decoder(video/image/picture sample decoder). The information decoder mayinclude the entropy decoder 310, and the sample decoder may include atleast one of the dequantizer 321, the inverse transformer 322, the adder340, the filter 350, the memory 360, the inter predictor 332, and theintra predictor 331.

The dequantizer 321 may dequantize the quantized transform coefficientsand output the transform coefficients. The dequantizer 321 may rearrangethe quantized transform coefficients in the form of a two-dimensionalblock form. In this case, the rearrangement may be performed based onthe coefficient scanning order performed in the encoding apparatus. Thedequantizer 321 may perform dequantization on the quantized transformcoefficients by using a quantization parameter (ex. quantization stepsize information) and obtain transform coefficients.

The inverse transformer 322 inversely transforms the transformcoefficients to obtain a residual signal (residual block, residualsample array).

The predictor may perform prediction on the current block and generate apredicted block including prediction samples for the current block. Thepredictor may determine whether intra prediction or inter prediction isapplied to the current block based on the information on the predictionoutput from the entropy decoder 310 and may determine a specificintra/inter prediction mode.

The predictor 320 may generate a prediction signal based on variousprediction methods described below. For example, the predictor may notonly apply intra prediction or inter prediction to predict one block butalso simultaneously apply intra prediction and inter prediction. Thismay be called combined inter and intra prediction (CIIP). In addition,the predictor may be based on an intra block copy (IBC) prediction modeor a palette mode for prediction of a block. The IBC prediction mode orpalette mode may be used for content image/video coding of a game or thelike, for example, screen content coding (SCC). The IBC basicallyperforms prediction in the current picture but may be performedsimilarly to inter prediction in that a reference block is derived inthe current picture. That is, the IBC may use at least one of the interprediction techniques described in the present disclosure. The palettemode may be considered as an example of intra coding or intraprediction. When the palette mode is applied, a sample value within apicture may be signaled based on information on the palette table andthe palette index.

The intra predictor 331 may predict the current block by referring tothe samples in the current picture. The referred samples may be locatedin the neighborhood of the current block or may be located apartaccording to the prediction mode. In the intra prediction, predictionmodes may include a plurality of non-directional modes and a pluralityof directional modes. The intra predictor 331 may determine theprediction mode applied to the current block by using a prediction modeapplied to a neighboring block.

The inter predictor 332 may derive a predicted block for the currentblock based on a reference block (reference sample array) specified by amotion vector on a reference picture. In this case, in order to reducethe amount of motion information transmitted in the inter predictionmode, motion information may be predicted in units of blocks,sub-blocks, or samples based on correlation of motion informationbetween the neighboring block and the current block. The motioninformation may include a motion vector and a reference picture index.The motion information may further include inter prediction direction(L0 prediction, L1 prediction, Bi prediction, etc.) information. In thecase of inter prediction, the neighboring block may include a spatialneighboring block present in the current picture and a temporalneighboring block present in the reference picture. For example, theinter predictor 332 may configure a motion information candidate listbased on neighboring blocks and derive a motion vector of the currentblock and/or a reference picture index based on the received candidateselection information. Inter prediction may be performed based onvarious prediction modes, and the information on the prediction mayinclude information indicating a mode of inter prediction for thecurrent block.

The adder 340 may generate a reconstructed signal (reconstructedpicture, reconstructed block, reconstructed sample array) by adding theobtained residual signal to the prediction signal (predicted block,predicted sample array) output from the predictor (including the interpredictor 332 and/or the intra predictor 331). If there is no residualfor the block to be processed, such as when the skip mode is applied,the predicted block may be used as the reconstructed block.

The adder 340 may be called reconstructor or a reconstructed blockgenerator. The generated reconstructed signal may be used for intraprediction of a next block to be processed in the current picture, maybe output through filtering as described below, or may be used for interprediction of a next picture.

Meanwhile, luma mapping with chroma scaling (LMCS) may be applied in thepicture decoding process.

The filter 350 may improve subjective/objective image quality byapplying filtering to the reconstructed signal. For example, the filter350 may generate a modified reconstructed picture by applying variousfiltering methods to the reconstructed picture and store the modifiedreconstructed picture in the memory 360, specifically, a DPB of thememory 360. The various filtering methods may include, for example,deblocking filtering, a sample adaptive offset, an adaptive loop filter,a bilateral filter, and the like.

The (modified) reconstructed picture stored in the DPB of the memory 360may be used as a reference picture in the inter predictor 332. Thememory 360 may store the motion information of the block from which themotion information in the current picture is derived (or decoded) and/orthe motion information of the blocks in the picture that have alreadybeen reconstructed. The stored motion information may be transmitted tothe inter predictor 260 so as to be utilized as the motion informationof the spatial neighboring block or the motion information of thetemporal neighboring block. The memory 360 may store reconstructedsamples of reconstructed blocks in the current picture and transfer thereconstructed samples to the intra predictor 331.

In the present document, the embodiments described in the filter 260,the inter predictor 221, and the intra predictor 222 of the encodingapparatus 200 may be the same as or respectively applied to correspondto the filter 350, the inter predictor 332, and the intra predictor 331of the decoding apparatus 300. The same may also apply to the unit 332and the intra predictor 331.

As described above, in video coding, prediction is performed to increasecompression efficiency. Through this, it is possible to generate apredicted block including prediction samples for a current block, whichis a block to be coded. Here, the predicted block includes predictionsamples in a spatial domain (or pixel domain). The predicted block isderived equally from the encoding device and the decoding device, andthe encoding device decodes information (residual information) on theresidual between the original block and the predicted block, not theoriginal sample value of the original block itself. By signaling to thedevice, image coding efficiency can be increased. The decoding apparatusmay derive a residual block including residual samples based on theresidual information, and generate a reconstructed block includingreconstructed samples by summing the residual block and the predictedblock, and generate a reconstructed picture including reconstructedblocks.

The residual information may be generated through transformation andquantization processes. For example, the encoding apparatus may derive aresidual block between the original block and the predicted block, andperform a transform process on residual samples (residual sample array)included in the residual block to derive transform coefficients, andthen, by performing a quantization process on the transformcoefficients, derive quantized transform coefficients to signal theresidual related information to the decoding apparatus (via abitstream). Here, the residual information may include locationinformation, a transform technique, a transform kernel, and aquantization parameter, value information of the quantized transformcoefficients etc. The decoding apparatus may performdequantization/inverse transformation process based on the residualinformation and derive residual samples (or residual blocks). Thedecoding apparatus may generate a reconstructed picture based on thepredicted block and the residual block. The encoding apparatus may alsodequantize/inverse transform the quantized transform coefficients forreference for inter prediction of a later picture to derive a residualblock, and generate a reconstructed picture based thereon.

In the present document, at least one of quantization/dequantizationand/or transform/inverse transform may be omitted. When thequantization/dequantization is omitted, the quantized transformcoefficient may be referred to as a transform coefficient. When thetransform/inverse transform is omitted, the transform coefficients maybe called coefficients or residual coefficients, or may still be calledtransform coefficients for uniformity of expression.

In the present document, a quantized transform coefficient and atransform coefficient may be referred to as a transform coefficient anda scaled transform coefficient, respectively. In this case, the residualinformation may include information on transform coefficient(s), and theinformation on the transform coefficient(s) may be signaled throughresidual coding syntax. Transform coefficients may be derived based onthe residual information (or information on the transformcoefficient(s)), and scaled transform coefficients may be derivedthrough inverse transform (scaling) on the transform coefficients.Residual samples may be derived based on an inverse transform(transform) of the scaled transform coefficients. This may beapplied/expressed in other parts of the present document as well.

Intra prediction may refer to prediction that generates predictionsamples for the current block based on reference samples in a picture towhich the current block belongs (hereinafter, referred to as a currentpicture). When intra prediction is applied to the current block,neighboring reference samples to be used for intra prediction of thecurrent block may be derived. The neighboring reference samples of thecurrent block may include samples adjacent to the left boundary of thecurrent block having a size of nW×nH and a total of 2×nH samplesneighboring the bottom-left, samples adjacent to the top boundary of thecurrent block and a total of 2×nW samples neighboring the top-right, andone sample neighboring the top-left of the current block. Alternatively,the neighboring reference samples of the current block may include aplurality of upper neighboring samples and a plurality of leftneighboring samples. In addition, the neighboring reference samples ofthe current block may include a total of nH samples adjacent to theright boundary of the current block having a size of nW×nH, a total ofnW samples adjacent to the bottom boundary of the current block, and onesample neighboring (bottom-right) neighboring bottom-right of thecurrent block.

However, some of the neighboring reference samples of the current blockmay not be decoded yet or available. In this case, the decoder mayconfigure the neighboring reference samples to use for prediction bysubstituting the samples that are not available with the availablesamples. Alternatively, neighboring reference samples to be used forprediction may be configured through interpolation of the availablesamples.

When the neighboring reference samples are derived, (i) the predictionsample may be derived based on the average or interpolation ofneighboring reference samples of the current block, and (ii) theprediction sample may be derived based on the reference sample presentin a specific (prediction) direction for the prediction sample among theperiphery reference samples of the current block. The case of (i) may becalled non-directional mode or non-angular mode and the case of (ii) maybe called directional mode or angular mode.

Furthermore, the prediction sample may also be generated throughinterpolation between the second neighboring sample and the firstneighboring sample located in a direction opposite to the predictiondirection of the intra prediction mode of the current block based on theprediction sample of the current block among the neighboring referencesamples. The above case may be referred to as linear interpolation intraprediction (LIP). In addition, chroma prediction samples may begenerated based on luma samples using a linear model. This case may becalled LM mode.

In addition, a temporary prediction sample of the current block may bederived based on filtered neighboring reference samples, and at leastone reference sample derived according to the intra prediction modeamong the existing neighboring reference samples, that is, unfilteredneighboring reference samples, and the temporary prediction sample maybe weighted-summed to derive the prediction sample of the current block.The above case may be referred to as position dependent intra prediction(PDPC).

In addition, a reference sample line having the highest predictionaccuracy among the neighboring multi-reference sample lines of thecurrent block may be selected to derive the prediction sample by usingthe reference sample located in the prediction direction on thecorresponding line, and then the reference sample line used herein maybe indicated (signaled) to the decoding apparatus, thereby performingintra-prediction encoding. The above case may be referred to asmulti-reference line (MRL) intra prediction or MRL based intraprediction.

In addition, intra prediction maybe performed based on the same intraprediction mode by dividing the current block into vertical orhorizontal subpartitions, and neighboring reference samples may bederived and used in the subpartition unit. That is, in this case, theintra prediction mode for the current block is equally applied to thesubpartitions, and the intra prediction performance may be improved insome cases by deriving and using the neighboring reference samples inthe subpartition unit. Such a prediction method may be called intrasub-partitions (ISP) or ISP based intra prediction.

The above-described intra prediction methods may be called an intraprediction type separately from the intra prediction mode. The intraprediction type may be called in various terms such as an intraprediction technique or an additional intra prediction mode. Forexample, the intra prediction type (or additional intra prediction mode)may include at least one of the above-described LIP, PDPC, MRL, and ISP.A general intra prediction method except for the specific intraprediction type such as LIP, PDPC, MRL, or ISP may be called a normalintra prediction type. The normal intra prediction type may be generallyapplied when the specific intra prediction type is not applied, andprediction may be performed based on the intra prediction mode describedabove. Meanwhile, post-filtering may be performed on the predictedsample derived as needed.

Specifically, the intra prediction procedure may include an intraprediction mode/type determination step, a neighboring reference samplederivation step, and an intra prediction mode/type based predictionsample derivation step. In addition, a post-filtering step may beperformed on the predicted sample derived as needed.

When intra prediction is applied, the intra prediction mode applied tothe current block may be determined using the intra prediction mode ofthe neighboring block. For example, the decoding apparatus may selectone of most probable mode (mpm) candidates of an mpm list derived basedon the intra prediction mode of the neighboring block (ex. left and/orupper neighboring blocks) of the current block based on the received mpmindex and select one of the other remaining intro prediction modes notincluded in the mpm candidates (and planar mode) based on the remainingintra prediction mode information. The mpm list may be configured toinclude or not include a planar mode as a candidate. For example, if thempm list includes the planar mode as a candidate, the mpm list may havesix candidates. If the mpm list does not include the planar mode as acandidate, the mpm list may have three candidates. When the mpm listdoes not include the planar mode as a candidate, a not planar flag (ex.intra_luma_not_planar_flag) indicating whether an intra prediction modeof the current block is not the planar mode may be signaled. Forexample, the mpm flag may be signaled first, and the mpm index and notplanar flag may be signaled when the value of the mpm flag is 1. Inaddition, the mpm index may be signaled when the value of the not planarflag is 1. Here, the mpm list is configured not to include the planarmode as a candidate does not is to signal the not planar flag first tocheck whether it is the planar mode first because the planar mode isalways considered as mpm.

For example, whether the intra prediction mode applied to the currentblock is in mpm candidates (and planar mode) or in remaining mode may beindicated based on the mpm flag (ex. Intra_luma_mpm_flag). A value 1 ofthe mpm flag may indicate that the intra prediction mode for the currentblock is within mpm candidates (and planar mode), and a value 0 of thempm flag may indicate that the intra prediction mode for the currentblock is not in the mpm candidates (and planar mode). The value 0 of thenot planar flag (ex. Intra_luma_not_planar_flag) may indicate that theintra prediction mode for the current block is planar mode, and thevalue 1 of the not planar flag value may indicate that the intraprediction mode for the current block is not the planar mode. The mpmindex may be signaled in the form of an mpm_idx or intra_luma_mpm_idxsyntax element, and the remaining intra prediction mode information maybe signaled in the form of a rem_intra_luma_pred_mode orintra_luma_mpm_remainder syntax element. For example, the remainingintra prediction mode information may index remaining intra predictionmodes not included in the mpm candidates (and planar mode) among allintra prediction modes in order of prediction mode number to indicateone of them. The intra prediction mode may be an intra prediction modefor a luma component (sample). Hereinafter, intra prediction modeinformation may include at least one of the mpm flag (ex.Intra_luma_mpm_flag), the not planar flag (ex.Intra_luma_not_planar_flag), the mpm index (ex. mpm_idx orintra_luma_mpm_idx), and the remaining intra prediction mode information(rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In the presentdocument, the MPM list may be referred to in various terms such as MPMcandidate list and candModeList. When MIP is applied to the currentblock, a separate mpm flag (ex. intra_mip_mpm_flag), an mpm index (ex.intra_mip_mpm_idx), and remaining intra prediction mode information (ex.intra_mip_mpm_remainder) for MIP may be signaled and the not planar flagis not signaled.

In other words, in general, when block splitting is performed on animage, a current block and a neighboring block to be coded have similarimage characteristics. Therefore, the current block and the neighboringblock have a high probability of having the same or similar intraprediction mode. Thus, the encoder may use the intra prediction mode ofthe neighboring block to encode the intra prediction mode of the currentblock.

For example, the encoder/decoder may configure a list of most probablemodes (MPM) for the current block. The MPM list may also be referred toas an MPM candidate list. Herein, the MPM may refer to a mode used toimprove coding efficiency in consideration of similarity between thecurrent block and neighboring block in intra prediction mode coding. Asdescribed above, the MPM list may be configured to include the planarmode or may be configured to exclude the planar mode. For example, whenthe MPM list includes the planar mode, the number of candidates in theMPM list may be 6. And, if the MPM list does not include the planarmode, the number of candidates in the MPM list may be 5.

The encoder/decoder may configure an MPM list including 5 or 6 MPMs.

In order to configure the MPM list, three types of modes can beconsidered: default intra modes, neighbor intra modes, and the derivedintra modes.

For the neighboring intra modes, two neighboring blocks, i.e., a leftneighboring block and an upper neighboring block, may be considered.

As described above, if the MPM list is configured not to include theplanar mode, the planar mode is excluded from the list, and the numberof MPM list candidates may be set to 5.

In addition, the non-directional mode (or non-angular mode) among theintra prediction modes may include a DC mode based on the average ofneighboring reference samples of the current block or a planar modebased on interpolation.

When inter prediction is applied, the predictor of the encodingapparatus/decoding apparatus may derive a prediction sample byperforming inter prediction in units of blocks. Inter prediction may bea prediction derived in a manner that is dependent on data elements (ex.sample values or motion information) of picture(s) other than thecurrent picture. When inter prediction is applied to the current block,a predicted block (prediction sample array) for the current block may bederived based on a reference block (reference sample array) specified bya motion vector on the reference picture indicated by the referencepicture index. Here, in order to reduce the amount of motion informationtransmitted in the inter prediction mode, the motion information of thecurrent block may be predicted in units of blocks, subblocks, or samplesbased on correlation of motion information between the neighboring blockand the current block. The motion information may include a motionvector and a reference picture index. The motion information may furtherinclude inter prediction type (L0 prediction, L1 prediction, Biprediction, etc.) information. In the case of inter prediction, theneighboring block may include a spatial neighboring block present in thecurrent picture and a temporal neighboring block present in thereference picture. The reference picture including the reference blockand the reference picture including the temporal neighboring block maybe the same or different. The temporal neighboring block may be called acollocated reference block, a co-located CU (colCU), and the like, andthe reference picture including the temporal neighboring block may becalled a collocated picture (colPic). For example, a motion informationcandidate list may be configured based on neighboring blocks of thecurrent block, and flag or index information indicating which candidateis selected (used) may be signaled to derive a motion vector and/or areference picture index of the current block. Inter prediction may beperformed based on various prediction modes. For example, in the case ofa skip mode and a merge mode, the motion information of the currentblock may be the same as motion information of the neighboring block. Inthe skip mode, unlike the merge mode, the residual signal may not betransmitted. In the case of the motion vector prediction (MVP) mode, themotion vector of the selected neighboring block may be used as a motionvector predictor and the motion vector of the current block may besignaled. In this case, the motion vector of the current block may bederived using the sum of the motion vector predictor and the motionvector difference.

The motion information may include L0 motion information and/or L1motion information according to an inter prediction type (L0 prediction,L1 prediction, Bi prediction, etc.). The motion vector in the L0direction may be referred to as an L0 motion vector or MVL0, and themotion vector in the L1 direction may be referred to as an L1 motionvector or MVL1. Prediction based on the L0 motion vector may be calledL0 prediction, prediction based on the L1 motion vector may be called L1prediction, and prediction based on both the L0 motion vector and the L1motion vector may be called bi-prediction. Here, the L0 motion vectormay indicate a motion vector associated with the reference picture listL0 (L0), and the L1 motion vector may indicate a motion vectorassociated with the reference picture list L1 (L1). The referencepicture list L0 may include pictures that are earlier in output orderthan the current picture as reference pictures, and the referencepicture list L1 may include pictures that are later in the output orderthan the current picture. The previous pictures may be called forward(reference) pictures, and the subsequent pictures may be called reverse(reference) pictures. The reference picture list L0 may further includepictures that are later in the output order than the current picture asreference pictures. In this case, the previous pictures may be indexedfirst in the reference picture list L0 and the subsequent pictures maybe indexed later. The reference picture list L1 may further includeprevious pictures in the output order than the current picture asreference pictures. In this case, the subsequent pictures may be indexedfirst in the reference picture list 1 and the previous pictures may beindexed later. The output order may correspond to picture order count(POC) order.

FIG. 4 exemplarily shows a hierarchical structure for a codedimage/video.

Referring to FIG. 4, coded image/video is divided into a video codinglayer (VCL) that handles the decoding process of the image/video anditself, a subsystem that transmits and stores the coded information, andNAL (network abstraction layer) in charge of function and presentbetween the VCL and the subsystem.

In the VCL, VCL data including compressed image data (slice data) isgenerated, or a parameter set including a picture parameter set (PSP), asequence parameter set (SPS), and a video parameter set (VPS) or asupplemental enhancement information (SEI) message additionally requiredfor an image decoding process may be generated.

In the NAL, a NAL unit may be generated by adding header information(NAL unit header) to a raw byte sequence payload (RBSP) generated in aVCL. In this case, the RBSP refers to slice data, parameter set, SEImessage, etc., generated in the VCL. The NAL unit header may include NALunit type information specified according to RBSP data included in thecorresponding NAL unit.

As shown in the figure, the NAL unit may be classified into a VCL NALunit and a Non-VCL NAL unit according to the RBSP generated in the VCL.The VCL NAL unit may mean a NAL unit that includes information on theimage (slice data) on the image, and the Non-VCL NAL unit may mean a NALunit that includes information (parameter set or SEI message) requiredfor decoding the image.

The above-described VCL NAL unit and Non-VCL NAL unit may be transmittedthrough a network by attaching header information according to the datastandard of the subsystem. For example, the NAL unit may be transformedinto a data format of a predetermined standard such as an H.266/VVC fileformat, a real-time transport protocol (RTP), a transport stream (TS),etc., and transmitted through various networks.

As described above, the NAL unit may be specified with the NAL unit typeaccording to the RBSP data structure included in the corresponding NALunit, and information on the NAL unit type may be stored and signaled inthe NAL unit header.

For example, the NAL unit may be classified into a VCL NAL unit type anda Non-VCL NAL unit type according to whether the NAL unit includesinformation (slice data) about an image. The VCL NAL unit type may beclassified according to the nature and type of pictures included in theVCL NAL unit, and the Non-VCL NAL unit type may be classified accordingto types of parameter sets.

The following is an example of the NAL unit type specified according tothe type of parameter set included in the Non-VCL NAL unit type.

-   -   APS (Adaptation Parameter Set) NAL unit: Type for NAL unit        including APS    -   DPS (Decoding Parameter Set) NAL unit: Type for NAL unit        including DPS    -   VPS(Video Parameter Set) NAL unit: Type for NAL unit including        VPS    -   SPS(Sequence Parameter Set) NAL unit: Type for NAL unit        including SPS    -   PPS(Picture Parameter Set) NAL unit: Type for NAL unit including        PPS    -   PH(Picture header) NAL unit: Type for NAL unit including PH

The aforementioned NAL unit types may have syntax information for theNAL unit type, and the syntax information may be stored and signaled ina NAL unit header. For example, the syntax information may benal_unit_type, and NAL unit types may be specified by a nal_unit_typevalue.

Meanwhile, as described above, one picture may include a plurality ofslices, and one slice may include a slice header and slice data. In thiscase, one picture header may be further added to a plurality of slices(a slice header and a slice data set) in one picture. The picture header(picture header syntax) may include information/parameters commonlyapplicable to the picture. In the present document, a slice may be mixedor replaced with a tile group. Also, in the present document, a sliceheader may be mixed or replaced with a tile group header.

The slice header (slice header syntax) may includeinformation/parameters that may be commonly applied to the slice. TheAPS (APS syntax) or the PPS (PPS syntax) may includeinformation/parameters that may be commonly applied to one or moreslices or pictures. The SPS (SPS syntax) may includeinformation/parameters that may be commonly applied to one or moresequences. The VPS (VPS syntax) may include information/parameters thatmay be commonly applied to multiple layers. The DPS (DPS syntax) mayinclude information/parameters that may be commonly applied to theoverall video. The DPS may include information/parameters related toconcatenation of a coded video sequence (CVS). The high level syntax(HLS) in the present document may include at least one of the APSsyntax, the PPS syntax, the SPS syntax, the VPS syntax, the DPS syntax,and the slice header syntax.

In the present document, the image/image information encoded from theencoding apparatus and signaled to the decoding apparatus in the form ofa bitstream includes not only partitioning related information in apicture, intra/inter prediction information, residual information,in-loop filtering information, etc., but also information included in aslice header, information included in the APS, information included inthe PPS, information included in an SPS, and/or information included inthe VPS.

Meanwhile, in order to compensate for a difference between an originalimage and a reconstructed image due to an error occurring in acompression coding process such as quantization, an in-loop filteringprocess may be performed on reconstructed samples or reconstructedpictures as described above. As described above, the in-loop filteringmay be performed by the filter of the encoding apparatus and the filterof the decoding apparatus, and a deblocking filter, SAO, and/or adaptiveloop filter (ALF) may be applied. For example, the ALF process may beperformed after the deblocking filtering process and/or the SAO processare completed. However, even in this case, the deblocking filteringprocess and/or the SAO process may be omitted.

Meanwhile, in order to increase coding efficiency, luma mapping withchroma scaling (LMCS) may be applied as described above. LMCS may bereferred to as a loop reshaper (reshaping). In order to increase codingefficiency, LMCS control and/or signaling of LMCS related informationmay be performed hierarchically.

FIG. 5 exemplarily illustrates a hierarchical structure of a CVSaccording to an embodiment of the present document. A coded videosequence (CVS) may include a sequence parameter set (SPS), a pictureparameter set (PPS), a tile group header, tile data, and/or CTU(s).Here, the tile group header and the tile data may be referred to as aslice header and slice data, respectively.

The SPS may include flags natively to enable tools to be used in CVS. Inaddition, the SPS may be referred to by the PPS including information onparameters that change for each picture. Each of the coded pictures mayinclude one or more coded rectangular domain tiles. The tiles may begrouped into raster scans forming tile groups. Each tile group isencapsulated with header information called a tile group header. Eachtile consists of a CTU comprising coded data. Here the data may includeoriginal sample values, prediction sample values, and its luma andchroma components (luma prediction sample values and chroma predictionsample values).

FIG. 6 illustrates an exemplary LMCS structure according to anembodiment of the present document. The LMCS structure 600 of FIG. 6includes an in-loop mapping part 610 of luma components based onadaptive piecewise linear (adaptive PWL) models and a luma-dependentchroma residual scaling part 620 for chroma components. Thedequantization and inverse transform 611, reconstruction 612, and intraprediction 613 blocks of the in-loop mapping part 610 representprocesses applied in the mapped (reshaped) domain. Loop filters 615,motion compensation or inter prediction 617 blocks of the in-loopmapping part 610, and reconstruction 622, intra prediction 623, motioncompensation or inter prediction 624, loop filters 625 block of thechroma residual scaling part 620 represent processes applied in theoriginal (non-mapped, non-reshaped) domain.

As illustrated in FIG. 6, when LMCS is enabled, at least one of theinverse mapping (reshaping) process 614, a forward mapping (reshaping)process 618, and a chroma scaling process 621 may be applied. Forexample, the inverse mapping process may be applied to a (reconstructed)luma sample (or luma samples or luma sample array) in a reconstructedpicture. The inverse mapping process may be performed based on apiecewise function (inverse) index of a luma sample. The piecewisefunction (inverse) index may identify the piece to which the luma samplebelongs. Output of the inverse mapping process is a modified(reconstructed) luma sample (or modified luma samples or modified lumasample array). The LMCS may be enabled or disabled at a level of a tilegroup (or slice), picture or higher.

The forward mapping process and/or the chroma scaling process may beapplied to generate the reconstructed picture. A picture may compriseluma samples and chroma samples. A reconstructed picture with lumasamples may be referred to as a reconstructed luma picture, and areconstructed picture with chroma samples may be referred to as areconstructed chroma picture. A combination of the reconstructed lumapicture and the reconstructed chroma picture may be referred to as areconstructed picture. The reconstructed luma picture may be generatedbased on the forward mapping process. For example, if an interprediction is applied to a current block, a forward mapping is appliedto a luma prediction sample derived based on a (reconstructed) lumasample in a reference picture. Because the (reconstructed) luma samplein the reference picture is generated based on the inverse mappingprocess, the forward mapping may be applied to the luma predictionsample thus a mapped (reshaped) luma prediction sample can be derived.The forward mapping process may be performed based on a piecewisefunction index of the luma prediction sample. The piecewise functionindex may be derived based on the value of the luma prediction sample orthe value of the luma sample in the reference picture used for interprediction. If an intra prediction (or an intra block copy (IBC)) isapplied to the current block, the forward mapping is not necessarybecause the inverse mapping process has not applied to the reconstructedsamples in the current picture yet. A (reconstructed) luma sample in thereconstructed luma picture is generated based on the mapped lumaprediction sample and a corresponding luma residual sample.

The reconstructed chroma picture may be generated based on the chromascaling process. For example, a (reconstructed) chroma sample in thereconstructed chroma picture may be derived based on a chroma predictionsample and a chroma residual sample (c_(res)) in a current block. Thechroma residual sample (c_(res)) is derived based on a (scaled) chromaresidual sample (c_(resScale)) and a chroma residual scaling factor(cScaleInv may be referred to as varScale) for the current block. Thechroma residual scaling factor may be calculated based on reshaped lumaprediction sample values for the current block. For example, the scalingfactor may be calculated based on an average luma value ave(Y′_(pred))of the reshaped luma prediction sample values Y′_(pred). For areference, in FIG. 6, the (scaled) chroma residual sample derived basedon the inverse transform/dequantization may be referred to asc_(resScale), and the chroma residual sample derived by performing the(inverse) scaling process to the (scaled) chroma residual sample may bereferred to as c_(re)s.

FIG. 7 illustrates an LMCS structure according to another embodiment ofthe present document. FIG. 7 is described with reference to FIG. 6.Here, the difference between the LMCS structure of FIG. 7 and the LMCSstructure 600 of FIG. 6 is mainly described. The in-loop mapping partand the luma-dependent chroma residual scaling part of FIG. 7 mayoperate the same as (similarly to) the in-loop mapping part 610 and theluma-dependent chroma residual scaling part 620 of FIG. 6.

Referring to FIG. 7, a chroma residual scaling factor may be derivedbased on luma reconstructed samples. In this case, an average luma value(avgYr) may be obtained (derived) based on the neighboring lumareconstructed samples outside the reconstructed block, not the innerluma reconstructed samples of the reconstructed block, and the chromaresidual scaling factor is derived based on the average luma value(avgYr). Here, the neighboring luma reconstructed samples may beneighboring luma reconstructed samples of the current block, or may beneighboring luma reconstructed samples of virtual pipeline data units(VPDUs) including the current block. For example, when intra predictionis applied to the target block, reconstructed samples may be derivedbased on prediction samples which are derived based on the intraprediction. In the other example, when inter prediction is applied tothe target block, the forward mapping is applied to prediction sampleswhich are derived based on the inter prediction, and reconstructedsamples are generated (derived) based on the reshaped (or forwardmapped) luma prediction samples.

The video/image information signaled through the bitstream may includeLMCS parameters (information on LMCS). LMCS parameters may be configuredas high level syntax (HLS, including slice header syntax) or the like.Detailed description and configuration of the LMCS parameters will bedescribed later. As described above, the syntax tables described in thepresent document (and the following embodiments) may beconfigured/encoded at the encoder end and signaled to the decoder endthrough a bitstream. The decoder may parse/decode information on theLMCS (in the form of syntax components) in the syntax tables. One ormore embodiments to be described below may be combined. The encoder mayencode the current picture based on the information about the LMCS andthe decoder may decode the current picture based on the informationabout the LMCS.

The in-loop mapping of luma components may adjust the dynamic range ofthe input signal by redistributing the codewords across the dynamicrange to improve compression efficiency. For luma mapping, a forwardmapping (reshaping) function (FwdMap) and an inverse mapping (reshaping)function (InvMap) corresponding to the forward mapping function (FwdMap)may be used. The FwdMap function may be signaled using a piece-wiselinear models, for example, the piece-wise linear model may have 16pieces or bins. The pieces may have the equal length. In one example,the InvMap function does not need to be signalled and is instead derivedfrom the FwdMap function. That is, the inverse mapping may be a functionof the forward mapping. For example, the inverse mapping function may bemathematically built as the symmetric function of the forward mapping asreflected by the line y=x.

An in-loop (luma) reshaping may be used to map input luma values(samples) to altered values in the reshaped domain. The reshaped valuesmay be coded and then mapped back into the original (un-mapped,un-reshaped) domain after reconstruction. To compensate for theinteraction between the luma signal and the chroma signal, chromaresidual scaling may be applied. In-loop reshaping is done by specifyinghigh level syntax for the reshaper model. The reshaper model syntax maysignal a piece-wise linear model (PWL model). For example, the reshapermodel syntax may signal a PWL model with 16 bins or pieces of equallengths. A forward lookup table (FwdLUT) and/or an inverse lookup table(InvLUT) may be derived based on the piece-wise linear model. Forexample, the PWL model pre-computes the 1024-entry forward (FwdLUT) andinverse (InvLUT) look up tables (LUT)s. As an example, when the forwardlookup table FwdLUT is derived, the inverse lookup table InvLUT may bederived based on the forward lookup table FwdLUT. The forward lookuptable FwdLUT may map the input luma values Yi to the altered values Yr,and the inverse lookup table InvLUT may map the altered values Yr to thereconstructed values Y′i. The reconstructed values Y′i may be derivedbased on the input luma values Yi.

In one example, the SPS may include the syntax of Table 1 below. Thesyntax of Table 1 may include sps_reshaper_enabled_flag as a toolenabling flag. Here, sps_reshaper_enabled_flag may be used to specifywhether the reshaper is used in a coded video sequence (CVS). That is,sps_reshaper_enabled_flag may be a flag for enabling reshaping in theSPS. In one example, the syntax of Table 1 may be a part of the SPS.

TABLE 1 Descriptor seq_parameter_set_rbsp( ) { sps_seq_parameter_set_idue(v) ... sps_reshaper_enabled_flag u(1) rbsp_trailing_bits( ) }

In one example, semantics on syntax elements sps_seq_parameter_set_idand sps_reshaper_enabled_flag may be as shown in Table 2 below.

TABLE 2 sps_seq_parameter_set_id provides an identifier for the SPS forreference by other syntax elements. sps_reshaper_enabled_flag equal to 1specifies that reshaper is used in the coded video sequence (CVS).sps_reshaper_enabled_flag equal to 0 specifies that reshaper is not usedin the CVS.

In one example, the tile group header or the slice header may includethe syntax of Table 3 or Table 4 below.

TABLE 3 Descriptor tile_group_header( ) { tile_group_pic_parameter_set_id ue(v) ...  if(num_tiles_in_tile_group_minus1 > 0 ) {   offset_len_minus1 ue(v)   for(i = 0; i < num_tiles_in_tile_group_minus1; i++ )   entry_point_offset_minus1[ i ] u(v)  }  if (sps_reshaper_enabled_flag ) {   tile_group_reshaper_model_present_flagu(1)   if ( tile_group_reshaper_model_present_flag )   tile_group_reshaper_model ( )   tile_group_reshaper_enable_flag u(1)  if ( tile_group_reshaper_enable_flag && (!( qtbtt_dual_tree_intra_flag&& tile_group_type == I ) ) )   tile_group_reshaper_chroma_residual_scale_flag u(1)  } byte_alignment( ) }

TABLE 4 Descriptor slice_header( ) {  slice_pic_parameter_set_id ue(v)...  if( num_tiles_in_slice_minus1 > 0 ) {   offset_len_minus1 ue(v)  for( i = 0; i < num_tiles_in_slice_minus1; i++ )   entry_point_offset_minus1[ i ] u(v)  }  if (sps_reshaper_enabled_flag ) {   slice_reshaper_model_present_flag u(1)  if ( slice_reshaper_model_present flag )    slice_reshaper_model ( )  slice_reshaper_enable_flag u(1)   if ( slice_reshaper_enable_flag &&(!( qtbtt_dual_tree_intra_flag && slice_type == I ) ) )   slice_reshaper_chroma_residual_scale_flag u(1)  }  byte_alignment( )}

Semantics of syntax elements included in the syntax of Table 3 or Table4 may include, for example, matters disclosed in the following tables.

TABLE 5 tile_group_reshaper_model_present_flag equal to 1 specifiestile_group_reshaper_model ( ) is present in tile group header.tile_group_reshaper_model_present_flag equal to 0 sp ecifiestile_group_reshaper_model( ) is not present in tile group header. Whentile_group _reshaper_model_present_flag is not present, it is inferredto be equal to 0. tile_group_reshaper_enabled_flag equal to 1 specifiesthat reshaper is enabled for the current tile group.tile_group_reshaper_enabled_flag equal to 0 specifies that reshaper is not enabled for the current tile group. Whentile_group_reshaper_enable_flag is not pr esent, it is inferred to beequal to 0. tile_group_reshaper_chroma_residual_scale_flag equal to 1specifies that chroma resid ual scaling is enabled for the current tilegroup. tile_group_reshaper_chroma_residual_s cale_flag equal to 0specifies that chroma residual scaling is not enabled for the curre nttile group. When tile_group_reshaper_chroma_residual_scale_flag is notpresent, it is inferred to be equal to 0.

TABLE 6 slice_reshaper_model_present_flag equal to 1 specifiesslice_reshaper_model( ) is prese nt in slice header,slice_reshaper_model_present_flag equal to 0 specifies slice_reshaper_model( ) is not present in slice header. Whenslice_reshaper_model_present_flag is not present, it is inferred to beequal to 0. slice_reshaper_enabled_flag equal to 1 specifies thatreshaper is enabled for the curre nt slice. slice_reshaper_enabled_flagequal to 0 specifies that reshaper is not enabled fo r the currentslice. When slice_reshaper_enable_flag is not present, it is inferred tobe equal to 0. slice_reshaper_chroma_residual_scale_flag equal to 1specifies that chroma residual scaling is enabled for the current slice.slice_reshaper_chroma_residual_scale_flag equal to 0 specifies thatchroma residual scaling is not enabled for the current slice. Whenslice_reshaper_chroma_residual_scale_flag is not present, it is inferredto be equal to 0.

As one example, once the flag enabling the reshaping (i.e.,sps_reshaper_enabled_flag) is parsed in the SPS, the tile group headermay parse additional data (i.e., information included in Table 5 or 6above) which is used to construct lookup tables (FwdLUT and/or InvLUT).In order to do this, the status of the SPS reshaper flag(sps_reshaper_enabled_flag) may be first checked in the slice header orthe tile group header. When sps_reshaper_enabled_flag is true (or 1), anadditional flag, i.e., tile_group_reshaper_model_present_flag (orslice_reshaper_model_present_flag) may be parsed. The purpose of thetile_group_reshaper_model_present_flag (orslice_reshaper_model_present_flag) may be to indicate the presence ofthe reshaping model. For example, whentile_group_reshaper_model_present_flag (orslice_reshaper_model_present_flag) is true (or 1), it may be indicatedthat the reshaper is present for the current tile group (or currentslice). When tile_group_reshaper_model_present_flag (orslice_reshaper_model_present_flag) is false (or 0), it may be indicatedthat the reshaper is not present for the current tile group (or currentslice).

If the reshaper is present and the reshaper is enabled in the currenttile group (or current slice), the reshaper model (i.e.,tile_group_reshaper_model( ) or slice_reshaper_model( )) may beprocessed. Further to this, an additional flag,tile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) may alsobe parsed. The tile_group_reshaper_enable_flag (orslice_reshaper_enable_flag) may indicate whether the reshaping model isused for the current tile group (or slice). For example, iftile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) is 0 (orfalse), it may be indicated that the reshaping model is not used for thecurrent tile group (or the current slice). Iftile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) is 1 (ortrue), it may be indicated that the reshaping model is used for thecurrent tile group (or slice).

As one example, tile_group_reshaper_model_present_flag (orslice_reshaper_model_present_flag) may be true (or 1) andtile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) may befalse (or 0). This means that the reshaping model is present but notused in the current tile group (or slice). In this case, the reshapingmodel can be used in the future tile groups (or slices). As anotherexample, tile_group_reshaper_enable_flag may be true (or 1) andtile_group_reshaper_model_present_flag may be false (or 0). In such acase, the decoder uses the reshaper from the previous initialization.

When the reshaping model (i.e., tile_group_reshaper_model( ) orslice_reshaper_model( )) and tile_group_reshaper_enable_flag (orslice_reshaper_enable_flag) are parsed, it may be determined (evaluated)whether conditions necessary for chroma scaling are present. The aboveconditions includes a condition 1 (the current tile group/slice has notbeen intra-coded) and/or a condition 2 (the current tile group/slice hasnot been partitioned into two separate coding quad tree structures forluma and chroma, i.e. the block structure for The current tilegroup/slice is not a dual tree structure). If the condition 1 and/or thecondition 2 are true and/or tile_group_reshaper_enable_flag (orslice_reshaper_enable_flag) is true (or 1), thentile_group_reshaper_chroma_residual_scale_flag (orslice_reshaper_chroma_residual_scale_flag) may be parsed. Whentile_group_reshaper_chroma_residual_scale_flag (orslice_reshaper_chroma_residual_scale_flag) is enabled (if 1 or true), itmay be indicated that chroma residual scaling is enabled for the currenttile group (or slice). Whentile_group_reshaper_chroma_residual_scale_flag (orslice_reshaper_chroma_residual_scale_flag) is disabled (if 0 or false),it may be indicated that chroma residual scaling is disabled for thecurrent tile group (or slice).

The purpose of the tile group reshaping model is to parse the data thatwould be necessary to construct the lookup tables (LUTs). These LUTs areconstructed on the idea that the distribution of an allowable range ofluma values can be divided into a plurality of bins (ex. 16 bins) whichcan be represented using a set of 16 PWL system of equations. Therefore,any luma value that lies within a given bin can be mapped to an alteredluma value.

FIG. 8 shows a graph representing an exemplary forward mapping. In FIG.8, five bins are illustrated exemplarily.

Referring to FIG. 8, the x-axis represents input luma values, and they-axis represents altered output luma values. The x-axis is divided into5 bins or slices, each bin of length L. That is, the five bins mapped tothe altered luma values have the same length. The forward lookup table(FwdLUT) may be constructed using data (i.e., reshaper data) availablefrom the tile group header, and thus mapping may be facilitated.

In one embodiment, output pivot points associated with the bin indicesmay be calculated. The output pivot points may set (mark) the minimumand maximum boundaries of the output range of the luma codewordreshaping. The calculation process of the output pivot points may beperformed by computing a piecewise cumulative distribution function(CDF) of the number of codewords. The output pivot range may be slicedbased on the maximum number of bins to be used and the size of thelookup table (FwdLUT or InvLUT). As one example, the output pivot rangemay be sliced based on a product between the maximum number of bins andthe size of the lookup table (size of LUT*maximum number of binindices). For example, if the product between the maximum number of binsand the size of the lookup table is 1024, the output pivot range may besliced into 1024 entries. This serration of the output pivot range maybe performed (applied or achieved) based on (using) a scaling factor. Inone example, the scaling factor may be derived based on Equation 1below.

$\begin{matrix}{{SF} = {{\left( {{y2} - {y1}} \right)*\left( {1{\operatorname{<<}{FP\_ PREC}}} \right)} + c}} & \left\lbrack {{Equation}1} \right\rbrack\end{matrix}$

In Equation 1, SF denotes a scaling factor, and y1 and y2 denote outputpivot points corresponding to each bin. Also, FP_PREC and c may bepredetermined constants. The scaling factor determined based on Equation1 may be referred to as a scaling factor for forward reshaping.

In another embodiment, with respect to inverse reshaping (inversemapping), for a defined range of the bins to be used (i.e., fromreshaper_model_min_bin_idx to reshape_model_max_bin_idx), the inputreshaped pivot points which correspond to the mapped pivot points of theforward LUT and the mapped inverse output pivot points (given by binindex under consideration*number of initial codewords) are fetched. Inanother example, the scaling factor SF may be derived based on Equation2 below.

$\begin{matrix}{{SF} = {\left( {{y2} - {y1}} \right)*\left( {1{\operatorname{<<}{FP\_ PREC}}} \right)/\left( {{x2} - {x1}} \right)}} & \left\lbrack {{Equation}2} \right\rbrack\end{matrix}$

In Equation 2, SF denotes a scaling factor, x1 and x2 denote input pivotpoints, and y1 and y2 denote output pivot points corresponding to eachpiece (bin) (output pivot points of the inverse mapping). Here, theinput pivot points may be pivot points mapped based on a forward lookuptable (FwdLUT), and the output pivot points may be pivot pointsinverse-mapped based on an inverse lookup table (InvLUT). Also, FP_PRECmay be a predetermined constant value. FP_PREC of Equation 2 may be thesame as or different from FP_PREC of Equation 1. The scaling factordetermined based on Equation 2 may be referred to as a scaling factorfor inverse reshaping. During inverse reshaping, partitioning of inputpivot points may be performed based on the scaling factor of Equation 2.The scaling factor SF is used to slice the range of input pivot points.Based on the partitioned input pivot points, bin indices in the rangefrom 0 to the minimum bin index (reshaper_model_min_bin_idx) and/or fromthe minimum bin index (reshaper_model_min_bin_idx) to the maximum binindex (reshape_model_max_bin_idx) are assigned the pivot values thatcorrespond to the minimum and maximum bin values.

Table 7 below shows the syntax of the reshaper model according to anembodiment. The reshaper model may be referred to as a reshaping modelor an LMCS model. Here, the reshaper model has been exemplarilydescribed as a tile group reshaper, but the present document is notnecessarily limited by this embodiment. For example, the reshaper modelmay be included in the APS, or the tile group reshaper model may bereferred to as a slice reshaper model or LMCS data. Also, the prefixreshaper_model or Rsp may be used interchangeably with lmcs. Forexample, in the following tables and the description below,reshaper_model_min_bin_idx, reshaper_model_delta_max_bin_idx,reshaper_model_max_bin_idx, RspCW, RsepDeltaCW may be usedinterchangeably with lmcs_min_bin_idx, lmcs_delta_cs_bin_idx, lmx,lmcs_delta_csDcselta_idx, lmx, respectively.

TABLE 7 Descriptor tile_group_reshaper_model ( ) { reshaper_model_min_bin_idx ue(v)  reshaper_model_delta_max_bin_idxue(v)  reshaper_model_bin_delta_abs_cw_prec_minus1 ue(v)  for ( i =reshaper_model_min_bin_idx; i <= reshaper_model_max_bin_idx; i++ ) {  reshape_model_bin_delta_abs_CW [ i ] u(v)   if (reshaper_model_bin_delta_abs_CW[ i ] ) >   0 )   reshaper_model_bin_delta_sign_CW_flag[ u(1)   i ]  } }

The semantics of syntax elements included in the syntax of Table 7 mayinclude, for example, matters disclosed in the following table.

TABLE 8 reshape_model_min_bin_idx specifies the minimum bin (or piece)index to be used in the reshaper construction process. The value ofreshape_model_min_bin_idx shall be in the range of 0 to MaxBinIdx,inclusive. The value of MaxBinIdx shall be equal to 15.reshape_model_delta_max_bin_idx specifies the maximum allowed bin (orpiece) index MaxBinIdx minus the maximum bin index to be used in thereshaper construction process. The value of reshape_model_max_bin_idx isset equal to MaxBinIdx − reshape_model_delta_max_bin_idx.reshaper_model_bin_delta_abs_cw_prec_minus1 plus 1 specifies the numberof bits used for the representation of the syntaxreshape_model_bin_delta_abs_CW[ i ]. reshape_model_bin_delta_abs_CW[ i ]specifies the absolute delta codeword value for the ith bin.reshaper_model_bin_delta_sign_CW_flag[ i ] specifies the sign ofreshape_model_bin_delta_abs_CW[ i ] as follows: - Ifreshape_model_bin_delta_sign_CW_flag [ i ] is equal to 0, thecorresponding variable RspDeltaCW[ i ] is a positive value. - Otherwise( reshape_model_bin_delta_sign_CW_flag[ i ] is not equal to 0 ), thecorresponding variable RspDeltaCW[ i ] is a negative value. The variableOrgCW is derived as follows: OrgCW = ( 1 << BitDepthY ) / 16 Whenreshape_model_bin_delta_sign_CW_flag[ i ] is not present, it is inferredto be equal to 0. The variable RspDeltaCW[ i ] = (1 −2*reshape_model_bin_delta_sign_CW [ i ]) *reshape_model_bin_delta_abs_CW [ i ]; The variable RspCW[ i ] is derivedas following steps: The variable OrgCW is set equal to (1 <<BitDepth_(γ) ) / ( MaxBinIdx + 1). - If reshaper_model_min_bin_idx < = i<= reshaper_model_max_bin_idx RspCW[ i ] = OrgCW + RspDeltaCW[ i ]. -Otherwise, RspCW[ i ] = 0. The value of RspCW[ i ] shall be in the rangeof (OrgCW>>3) to (OrgCW<<3 − 1), inclusive. The variables InputPivot[ i] with i in the range of 0 to MaxBinIdx + 1, inclusive are derived asfollows: InputPivot[ i ] = i * OrgCW  The variable ReshapePivot[ i ]with i in the range of 0 to MaxBinIdx + 1, inclusive, the  variableScaleCoef[ i ] and InvScaleCoeff[ i ]with i in the range of 0 toMaxBinIdx , inclusive,  are derived as follows:   shiftY = 11  ReshapePivot[ 0 ] = 0;   for( i = 0; i <= MaxBinIdx ; i++) {  ReshapePivot[ i + 1 ] = ReshapePivot[ i ] + RspCW[ i ]   ScaleCoef[ i] = ( RspCW[ i ] * (1 << shiftY) + (1 << (Log2(OrgCW) − 1))) >>  (Log2(OrgCW))   if ( RspCW[ i ] == 0 )    InvScaleCoeff[ i ] = 0  else    InvScaleCoeff[ i ] = OrgCW * (1 << shiftY) / RspCW[ i ]   } The variable ChromaScaleCoef[ i ] with i in the range of 0 to MaxBinIdx, inclusive, are  derived as follows:  if ( lmcsCW[ i ] = = 0 )  ChromaScaleCoeff[ i ] = (1 << 11)  else   ChromaScaleCoeff[ i ] =InvScaleCoeff[ i ]

The inverse mapping process for the luma sample according to the presentdocument may be described in a form of the standard document as shown inthe table below.

TABLE 9 Inverse mapping process for a luma sample Input to this processis a luma sample lumaSample. Output of this process is a modified lumasample invLumaSample. The value of invLumaSample is derived asfollows: -  If slice_lmcs_enabled_flag of the slice that contains theluma sample lumaSample is equal  to 1, the following ordered stepsapply: 1. The variable idxYInv is derived by invoking the identificationof piece-wise function index process for a luma sample as specified inclause 8.8.2.3 with lumaSample as the input and idxYInv as the output.2. The variable invSample is derived as follows: invSample = InputPivot[idxYInv ] + (InvScaleCoeff[ idxYInv ] *  (lumaSample − LmcsPivot[idxYInv ]) + ( 1 << 10 ) ) >> 11 3. The inverse mapped luma sampleinvLumaSample is derived as follows: invLumaSample = Clip1Y( invSample) -  Otherwise, invLumaSample is set equal to lumaSample.

Identification of a piecewise function index process for a luma sampleaccording to the present document may be described in a form of thestandard document as shown in the table below. In Table 10, idxYInv maybe referred to as an inverse mapping index, and the inverse mappingindex may be derived based on reconstructed luma samples (lumaSample).

TABLE 10 Identification of piecewise function index process for a lumasample Input to this process is a luma sample lumaSample. Output of thisprocess is an index idxYInv identifing the piece to which the lumasample lumaSample belongs. The variable idxYInv is derived as follows:if ( lumaSample < LmcsPivot[ lmcs_min_bin_idx + 1 ] )  idxYInv =lmcs_min_bin_idx else if ( lumaSample >= LmcsPivot[ LmcsMaxBinIdx ] ) idxYInv = LmcsMaxBinIdx else {  for( idxYInv = lmcs_min_bin_idx;idxYInv < LmcsMaxBinIdx; idxYInv++ ) {   if( lumaSample < LmcsPivot [idxYInv + 1 ] )    break  } }

Luma mapping may be performed based on the above-described embodimentsand examples, and the above-described syntax and components includedtherein may be merely exemplary representations, and embodiments in thepresent document are not limited by the above-mentioned tables orequations. Hereinafter, a method for performing chroma residual scaling(scaling for chroma components of residual samples) based on lumamapping is described.

The (luma-dependent) Chroma residual scaling is designed to compensatefor the interaction between the luma signal and its corresponding chromasignals. For example, whether chroma residual scaling is enabled or notis also signalled at the tile group level. In one example, if lumamapping is enabled and if dual tree partition (also known as separatechroma tree) is not applied to the current tile group, an additionalflag is signalled to indicate if the luma-dependent chroma residualscaling is enabled or not. In other example, when luma mapping is notused, or when dual tree partition is used in the current tile group,luma-dependent chroma residual scaling is disabled. In another example,the luma-dependent chroma residual scaling is always disabled for thechroma blocks whose area is less than or equal to 4.

The chroma residual scaling may be based on an average value of acorresponding luma prediction block (a luma component of a predictionblock to which an intra prediction mode and/or an inter prediction modeis applied). Scaling operations at the encoder end and/or the decoderside may be implemented with fixed-point integer arithmetic based onEquation 3 below.

$\begin{matrix}{{c’} = {{{sign}(c)}*\left( {\left( {{{abs}(c)*s} + {2{CSCALE\_ FP}{\_ PREC}} - 1} \right)\operatorname{>>}{CSCALE\_ FP\_ PREC}} \right)}} & \left\lbrack {{Equation}3} \right\rbrack\end{matrix}$

In Equation 3, c denotes a chroma residue (chroma residual sample,chroma component of residual sample), c′ denotes a scaled chromaresidual sample (scaled chroma component of a residual sample), sdenotes a chroma residual scaling factor, CSCALE_FP_PREC denotes a(predefined) constant value to specify precision, and for example,CSCALE_FP_PREC may be 11.

FIG. 9 is a flowchart illustrating a method for deriving a chromaresidual scaling index according to an embodiment of the presentdocument. The method in FIG. 9 may be performed based on FIG. 6, andtables, equations, variables, arrays, and functions included in thedescription related to FIG. 6.

In the step S910, it may be determined whether the prediction mode forthe current block is the intra prediction mode or the inter predictionmode based on the prediction mode information. If the prediction mode isthe intra prediction mode, the current block or prediction samples ofthe current block are considered to be already in the reshaped (mapped)region. If the prediction mode is the inter prediction mode, the currentblock or the prediction samples of the current block are considered tobe in the original (unmapped, non-reshaped) region.

In the step S920, when the prediction mode is the intra prediction mode,an average of the current block (or luma prediction samples of thecurrent block) may be calculated (derived). That is, the average of thecurrent block in the already reshaped area is calculated directly. Theaverage may also be referred to as an average value, a mean, or a meanvalue.

In the step S921, when the prediction mode is the inter prediction mode,forward reshaping (forward mapping) may be performed (applied) on theluma prediction samples of the current block. Through forward reshaping,luma prediction samples based on the inter prediction mode may be mappedfrom the original region to the reshaped region. In one example, forwardreshaping of the luma prediction samples may be performed based on thereshaping model described with Table 7 above.

In the step S922, an average of the forward reshaped (forward mapped)luma prediction samples may be calculated (derived). That is, anaveraging process for the forward reshaped result may be performed.

In the step S930, a chroma residual scaling index may be calculated.When the prediction mode is the intra prediction mode, the chromaresidual scaling index may be calculated based on the average of theluma prediction samples. When the prediction mode is the interprediction mode, the chroma residual scaling index may be calculatedbased on an average of forward reshaped luma prediction samples.

In an embodiment, the chroma residual scaling index may be calculatedbased on a for loop syntax. The table below shows an exemplary for loopsyntax for deriving (calculating) the chroma residual scaling index.

TABLE 11 for( idxS = 0, idxFound = 0; idxS <= MaxBinIdx; idxS++ ) {  if((S < ReshapePivot[ idxS + 1 ] )  {   idxFound = 1   break;  } }

In Table 11, idxS represents the chroma residual scaling index, idxFoundrepresents an index identifying whether the chroma residual scalingindex satisfying the condition of the if statement is obtained, Srepresents a predetermined constant value, and MaxBinIdx represents themaximum allowable bin index. ReshapPivot[idxS+1] may be derived based onTables 7 and/or 8 described above.

In an embodiment, the chroma residual scaling factor may be derivedbased on the chroma residual scaling index. Equation 4 is an example forderiving the chroma residual scaling factor.

$\begin{matrix}\left. {{s = {ChromaScaleCoe{f\lbrack}}}{idxS}} \right\rbrack & \left\lbrack {{Equation}4} \right\rbrack\end{matrix}$

In Equation 4, s represents the chroma residual scaling factor, andChromaScaleCoef may be a variable (or array) derived based on Tables 7and/or 8 described above.

As described above, the average luma value of the reference samples maybe obtained, and the chroma residual scaling factor may be derived basedon the average luma value. As described above, the chroma componentresidual sample may be scaled based on the chroma residual scalingfactor, and the chroma component reconstruction sample may be generatedbased on the scaled chroma component residual sample.

In one embodiment of the present document, a signaling structure forefficiently applying the above-described LMCS is proposed. According tothis embodiment of the present document, for example, LMCS data may beincluded in HLS (i.e., an APS), and through the header information(i.e., picture header, slice header) that is a lower level of the APS,an LMCS model (reshaper model) may be adaptively derived by signalingthe ID of the APS, which is referred to the header information. The LMCSmodel may be derived based on LMCS parameters. Also, for example, aplurality of APS IDs may be signaled through the header information, andthrough this, different LMCS models may be applied in units of blockswithin the same picture/slice.

In one embodiment according to the present document, a method forefficiently performing an operation required for LMCS is proposed.According to the semantics described above in Table 8, a divisionoperation by the piece length lmcsCW[i] (also noted as RspCW[i] in thepresent document) is required to derive InvScaleCoeff[i]. The piecelength of the inverse mapping may not be power of 2, that means thedivision cannot be performed by bit shifting.

For example, calculating InvScaleCoeff may require up to 16 divisionsper slice. According to Table 8 described above, for 10 bit coding, therange of lmcsCW[i] is from 8 to 511, so to implement the divisionoperation by lmcsCW[i] using the LUT, the size of the LUT must be 504.Also, for 12 bit coding, the range of lmcsCW[i] is from 32 to 2047, sothe LUT size needs be 2016 to implement the division operation bylmcsCW[i] using the LUT. That is, division is expensive in hardwareimplementation, therefore it is desirable to avoid division if possible.

In one aspect of this embodiment, lmcsCW[i] may be constrained to bemultiple of a fixed number (or a predetermined number or apre-determined number). Accordingly, a lookup table (LUT) (capacity orsize of the LUT) for the division may be reduced. For example, iflmcsCW[i] becomes multiple of 2, the size of the LUT to replace divisionprocess may be reduced by half.

In another aspect of this embodiment, it is proposed that for codingwith higher internal bit depth coding, on the top of existingconstraints “The value of lmcsCW[i] shall be in the range of (OrgCW>>3)to (OrgCW<<3−1)”, further constrain lmcsCW[i] to be multiple of1<<(BitDepthY−10) if coding bit depth is higher than 10. Here, BitDepthYmay be the luma bit depth. Accordingly, the possible number of lmcsCW[i]would not vary with the coding bitdepth, and the size of LUT needed tocalculate the InvScaleCoeff do not increase for higher coding bitdepth.For example, for 12-bit internal coding bitdepth, limit the values oflmcsCW[i] being multiple of 4, then the LUT to replace division processwill be the same as what is to be used for 10-bit coding. This aspectcan be implemented alone, but it can also be implemented in combinationwith the above-mentioned aspect.

In another aspect of this embodiment, lmcsCW[i] may be constrained to anarrower range. For example, lmcsCW[i] may be constrained within therange from (OrgCW>>1) to (OrgCW<<1)−1. Then for 10 bit coding, the rangeof lmcsCW[i] may be [32, 127], and therefore it only needs a LUT havinga size of 96 to calculate InvScaleCoeff.

In another aspect of the present embodiment, lmcsCW[i] may beapproximated to closest numbers being power of 2, and used that in thereshaper design. Accordingly, the division in the inverse mapping can beperformed (and replaced) by bit shifting.

In one embodiment according to the present document, constraint of theLMCS codeword range is proposed. According to Table 8 described above,the values of the LMCS codewords are in the range from (OrgCW>>3) to(OrgCW<<3)−1. This codword range is too wide. It may result in visualartifact issue when there are large differences between RspCW [i] andOrgCW.

According to one embodiment according to the present document, it isproposed to constrain the codeword of the LMCS PWL mapping to a narrowrange. For example, the range of lmcsCW[i] may be in the range(OrgCW>>1) to (OrgCW<<1)−1.

In one embodiment according to the present document, use of a singlechroma residual scaling factor is proposed for chroma residual scalingin LMCS. The existing method for deriving the chroma residual scalingfactor uses the average value of the corresponding luma block andderives the slope of each piece of the inverse luma mapping as thecorresponding scaling factor. In addition, the process to identify thepiecewise index requires the availability of the corresponding lumablock, it results in latency problem. This is not desirable for hardwareimplementation. According to this embodiment of the present document,scaling in a chroma block may not depend on a luma block value, and itmay not be necessary to identify a piecewise index. Therefore, thechroma residual scaling process in the LMCS can be performed without alatency issue.

In an embodiment according to the present document, a single chromascaling factor may be derived at both the encoder and the decoder basedon the luma LMCS information. When the LMCS luma model is received, thechroma residual scaling factor may be updated. For example, when theLMCS model is updated, a single chroma residual scaling factor may beupdated.

The table below shows an example for obtaining a single chroma scalingfactor according to the present embodiment.

TABLE 12 Sum = 0; for( i = lmcs_min_bin_idx ; i <= lmcs_max_bin_idx ;i++ ) {  sum += InvScaleCoeff[ i ] } ChromaScaleCoeff= sum / (lmcs_max_bin_idx− lmcs_min_bin_idx +1);

Referring to Table 12, a single chroma scaling factor (ex.ChromaScaleCoeff or ChromaScaleCoeffSingle) may be obtained by averagingthe inverse luma mapping slopes of all pieces within thelmcs_min_bin_idx and lmcs_max_bin_idx.

FIG. 10 illustrates a linear fitting of pivot points according to anembodiment of the present document. In FIG. 10, pivot points P1, Ps, P2are shown. The following embodiments or examples thereof is be describedwith reference to FIG. 10.

In an example of this embodiment, a single chroma scaling factor may beobtained based on a linear approximation of the luma PWL mapping betweenthe pivot points lmcs_min_bin_idx and lmcs_max_bin_idx+1. That is, theinverse gradient of the linear mapping may be used as the chromaresidual scaling factor. For example, the linear line 1 of FIG. 10 maybe a straight line connecting the pivot points P1 and P2. Referring toFIG. 10, in P1, the input value is x1 and the mapped value is 0, and inP2, the input value is x2 and the mapped value is y2. The inverse slope(inverse scale) of linear line 1 is (x2−x1)/y2, and a single chromascaling factor ChromaScaleCoeffSingle may be calculated based on theinput values and mapped values of pivot points P1, P2, and the followingequation.

$\begin{matrix}{{ChromaScaleCoeffSingle} = {\left( {{x2} - {x1}} \right)*\left( {1{\operatorname{<<}{CSCALE\_ FP\_ PREC}}} \right)/y2}} & \left\lbrack {{Equation}5} \right\rbrack\end{matrix}$

In Equation 5, CSCALE_FP_PREC represents a shift factor, and, forexample, CSCALE_FP_PREC may be a predetermined constant. In one example,CSCALE_FP_PREC may be 11.

In another example according to this embodiment, referring to FIG. 10,the input value at the pivot point Ps is min_bin_idx+1 and the mappedvalue is ys. Accordingly, the inverse slope (inverse scale) of linearline 1 can be calculated as (xs−x1)/ys, and the single chroma scalingfactor ChromaScaleCoeffSingle may be calculated based on the inputvalues and mapped values of pivot points P1, Ps, and the followingequation.

$\begin{matrix}{{ChromaScaleCoeffSingle} = {\left( {{xs} - {x1}} \right)*\left( {1{\operatorname{<<}{CSCALE\_ FP\_ PREC}}} \right)/{ys}}} & \left\lbrack {{Equation}6} \right\rbrack\end{matrix}$

In Equation 6, CSCALE_FP_PREC represents a shift factor (a factor forbit shifting), for example, CSCALE_FP_PREC may be a predeterminedconstant. In one example, CSCALE_FP_PREC may be 11, and bit shifting forinverse scale may be performed based on CSCALE_FP_PREC.

In another example according to this embodiment, a single chromaresidual scaling factor may be derived based on a linear approximationline. An example for deriving a linear approximation line may include alinear connection of pivot points (i.e., lmcs_min_bin_idx,lmcs_max_bin_idx+1). For example, the linear approximation result may berepresented by codewords of PWL mapping. The mapped value y2 at P2 maybe the sum of the codewords of all bins (fragments), and the difference(x2−x1) between the input value at P2 and the input value at P1 isOrgCW*(lmcs_max_bin_idx−lmcs_min_bin_idx+1) (OrgCW see Table 8 above).The table below shows an example of obtaining a single chroma scalingfactor according to the above-described embodiment.

TABLE 13 Sum = 0; for( i = lmcs_min_bin_idx ; i <= lmcs_max_bin_idx ;i++ ) {  sum += lmcsCW[ i ] } ChromaScaleCoeffSingle = OrgCW *(lmcs_max_bin_idx− lmcs_min_bin_idx +1)   * (1 << CSCALE_FP_PREC) / sum;

Referring to Table 13, the single chroma scaling factor (ex.ChromaScaleCoeffSingle) may be obtained from two pivot points (i.e.,lmcs_min_bin_idx, lmcs_max_bin_idx). For example, the inverse slope oflinear mapping may be used as the chroma scaling factor.

In another example of this embodiment, the single chroma scaling factormay be obtained by linear fitting of pivot points to minimize an error(or mean square error) between the linear fitting and the existing PWLmapping. This example may be more accurate than simply connecting thetwo pivot points at lmcs_min_bin_idx and lmcs_max_bin_idx. There aremany ways to find the optimal linear mapping and such example isdescribed below.

In one example, parameters b1 and b0 of a linear fitting equationy=b1*x+b0 for minimizing the sum of least square error may be calculatedbased on Equations 5 and/or 6 below.

$\begin{matrix}{{b1} = \frac{\sum\limits_{1}^{n}{\left( {x_{i} - \overset{\_}{x}} \right)\left( {y_{i} - \overset{\_}{y}} \right)}}{\sum\limits_{1}^{n}\left( {x_{i} - \overset{\_}{x}} \right)^{2}}} & \left\lbrack {{Equation}7} \right\rbrack\end{matrix}$ $\begin{matrix}{{b0} = {\overset{\_}{y} - {b_{1}\overset{\_}{x}}}} & \left\lbrack {{Equation}8} \right\rbrack\end{matrix}$

In Equation 7 and 8, x is an original luma value, and y are a reshapedluma value. In more detail, x and y are the mean of x and y, and x_(i)and y_(i) represent values of the i-th pivot points.

Referring to FIG. 10, another simple approximation to identify thelinear mapping is given as:

-   -   Obtain linear line 1 by connecting the pivot points of the PWL        mapping at lmcs_min_bin_idx and lmcs_max_bin_idx+1, and compute        lmcs_pivots_linear[i] on the linear line with input values that        are multiples of OrgCW.    -   Sum up the differences between the mapped values of the pivot        points by using the linear line1 and PWL mapping.    -   Get the average difference avgDiff.    -   Adjust the last pivot point of the linear line according to the        average difference, e.g., 2*avgDiff    -   Use the inverse slope of the adjusted linear line as the chroma        residual scale.

According to the above-described linear fitting, the chroma scalingfactor (i.e., the inverse slope of forward mapping) may be derived(obtained) based on Equation 7 or 8 below.

$\begin{matrix}{{{ChromaScaleCoeffSingle} = {{OrgCW}*\left( {1{\operatorname{<<}{CSCALE\_ FP\_ PREC}}} \right)/}}{{lmcs\_ pivots}{{\_ linear}\left\lbrack {{{lmcs\_ min}{\_ bin}{\_ idx}} + 1} \right\rbrack}}} & \left\lbrack {{Equation}9} \right\rbrack\end{matrix}$ $\begin{matrix}{{{ChromaScaleCoeffSingle} = {{OrgCW}*\left( {{{lmcs\_ max}{\_ bin}{\_ idx}} - {{lmcs\_ bin}{\_ idx}} + 1} \right)*\left( {1{\operatorname{<<}{CSCALE\_ FP\_ PREC}}} \right)/}}{{lmcs\_ pivots}{{\_ linear}\left\lbrack {{{lmcs\_ max}{\_ bin}{\_ idx}} + 1} \right\rbrack}}} & \left\lbrack {{Equation}10} \right\rbrack\end{matrix}$

In the above-described equations, lmcs_pivots_lienar[i] may be mappedvalues of linear mapping. With linear mapping, all pieces of PWL mappingbetween minimum and maximum bin indices may have the same LMCS codeword(lmcsCW). That is, lmcs_pivots_linear[lmcs_min_bin_idx+1] may be thesame as lmcsCW[lmcs_min_bin_idx].

Also, in Equations 9 and 10, CSCALE_FP_PREC represents a shift factor (afactor for bit shifting), for example, CSCALE_FP_PREC may be apredetermined constant. In one example, CSCALE_FP_PREC may be 11.

With a single chroma residual scaling factor (ChromaScaleCoeffSingle),there is no need to calculate the average of the corresponding lumablock and there is no need to find the index in the PWL linear mapping.Accordingly, the efficiency of coding using chroma residual scaling maybe increased.

In another embodiment of this document, the encoder may determineparameters relating to a single chroma scaling factor and signal theparameters to the decoder. Through signaling, the encoder may makeavailable other information available at the encoder to derive a chromaresidual scaling factor. This embodiment aims to eliminate the chromaresidual scaling latency problem.

For example, a procedure for identifying a linear mapping used todetermine the chroma residual scaling factor may be given as follows:

-   -   Obtain linear line 1 by connecting the pivot points of the PWL        mapping at lmcs_min_bin_idx and lmcs_max_bin_idx+1, and compute        lmcs_pivots_linear[i] on the linear line with input values that        are multiples of OrgCW.    -   Obtain a weighted sum of differences between mapped values of        pivot points using pivot points of linear line 1 and luma PWL        mapping    -   Get weighted average difference (avgDiff)    -   Adjust the last pivot point of linear line 1 according to the        weighted mean difference (i.e., 2*avgDiff)    -   Use the inverse slope of the adjusted linear line as the chroma        residual scale

The table below shows examples of syntaxes for signaling the y value forchroma scaling factor derivation.

TABLE 14 Descriptor lmcs_data ( ) {  lmcs_min_bin_idx ue(v) lmcs_delta_max_bin_idx ue(v)  lmcs_delta_cvv_prec_minus1 ue(v)  for ( i= lmcs_min_bin_idx; i <= LmcsMaxBinIdx, i++ )  {   lmcs_delta_abs_cw[ i] u(v)   if ( lmcs_delta_abs_cw[ i ] ) > 0 )    lmcs_delta_sign_cw_flag[i ] u(1)  }  lmcs_chroma_scale u(v) }

In Table 14, the syntax element lmcs_chroma_scale may specify a singlechroma (residual) scaling factor used for LMCS chroma residual scaling(ChromaScaleCoeffSingle=lmcs_chroma_scale). That is, information on thechroma residual scaling factor may be directly signaled, and thesignaled information may be derived as the chroma residual scalingfactor. In other words, the value of the signaled information on thechroma residual scaling factor may be (directly) derived as the value ofthe single chroma residual scaling factor. Here, the syntax elementlmcs_chroma_scale may be signaled together with other LMCS data (i.e., asyntax element related to the absolute value and the sign of thecodeword etc.).

Alternatively, the encoder may signal to the decoder only parametersnecessary to derive the chroma residual scaling factor. In order toderive the chroma residual scaling factor in the decoder, an input valuex and a mapped value y are needed. Since the x value represents the binlength, it is a value already known to the decoder side and does notneed to be signaled. After all, only the y value needs to be signaled inorder to derive the chroma residual scaling factor. Here, the y valuemay be a mapped value of any pivot point in the linear mapping (e.g.mapped values of P2 or Ps in FIG. 10).

The following tables show examples of signaling mapped values forderiving a chroma residual scaling factor.

TABLE 15 Descriptor lmcs_data ( ) {  lmcs_min_bin_idx ue(v) lmcs_delta_max_bin_idx ue(v)  lmcs_delta_cw_prec_minus1 ue(v)  for ( i= lmcs_min_bin_idx; i <= LmcsMaxBmIdx;  i++ ) {   lmcs_dclta_abs_cw[ i ]u(v)   if ( lmcs_dclta_abs_cw[ i ] ) > 0 )    lmcs_delta_sign_cw_flag[ i] u(1)  }   lmcs_cw_linear u(v) }

TABLE 16 Descriptor lmcs_data ( ) {  lmcs_min_bin_idx ue(v) lmcs_delta_max_bin_idx ue(v)  lmcs_delta_cw_prec_minus1 ue(v)  for ( i= lmcs_min_bin_idx; i <= LmcsMaxBinIdx;  i++ ) {   lmcs_delta_abs_cw[ i] u(v)   if ( lmcs_delta_abs_cw[ i ] ) > 0 )    lmcs_delta_sign_cw_flag[i ] u(1)  }   lmcs_delta_abs_cw_linear u(v)   if (lmcs_delta_abs_cw_linear ) > 0 )    lmcs_delta_sign_cw_linear_flag u(1)}

One of the syntaxes of Tables 15 and 16 described above may be used tosignal the y value at any linear pivot points specified by the encoderand decoder. That is, the encoder and the decoder may derive the y valueusing the same syntax.

First, an embodiment according to Table 15 will be described. In Table15, lmcs_cw_linear may indicate a value mapped to Ps or P2. That is, inthe embodiment according to Table 15, a fixed number may be signaledthrough lmcs_cw_linear.

In an example according to this embodiment, if lmcs_cw_linear representsa value mapped to one bin (i.e., lmcs_pivots_linear[lmcs_min_bin_idx+1]in Ps of FIG. 10), the chroma scaling factor may be derived based on thefollowing equation.

$\begin{matrix}{{ChromaScaleCoeffSingle} = {{OrgCW}*\left( {1{\operatorname{<<}{CSCALE\_ FP\_ PREC}}} \right)/{lmcs\_ cw}{\_ linear}}} & \left\lbrack {{Equation}11} \right\rbrack\end{matrix}$

In another example according to this embodiment, if lmcs_cw_linearrepresents lmcs_max_bin_idx+1 (i.e.lmcs_pivots_linear[lmcs_max_bin_idx+1] in P2 of FIG. 10), the chromascaling factor may be derived based on the following equation.

$\begin{matrix}{{ChromaScaleCoeffSingle} = {{OrgCW}*\left( {{{lmcs\_ max}{\_ bin}{\_ idx}} - {{lmcs\_ max}{\_ bin}{\_ idx}} + 1} \right)*\left( {1{\operatorname{<<}{CSCALE\_ FP\_ PREC}}} \right)/{lmcs\_ cw}{\_ linear}}} & \left\lbrack {{Equation}12} \right\rbrack\end{matrix}$

In the above-described equations, CSCALE_FP_PREC represents a shiftfactor (a factor for bit shifting), for example, CSCALE_FP_PREC may be apredetermined constant. In one example, CSCALE_FP_PREC may be 11.

Next, an embodiment according to Table 16 is described. In thisembodiment, lmcs_cw_linear may be signaled as a fixed number associateddelta value (i.e., lmcs_delta_abs_cw_linear,lmcs_delta_sign_cw_linear_flag). In an example of this embodiment, whenlmcs_cw_linear represents a mapped value inlmcs_pivots_linear[lmcs_min_bin_idx+1] (i.e., Ps of FIG. 10),lmcs_cw_linear_delta and lmcs_cw_linear may be derived based on thefollowing equations.

$\begin{matrix}{{{lmcs\_ cw}{\_ linear}{\_ delta}} = {\left( {1 - {2*{lmcs\_ delta}{\_ sign}{\_ cw}{\_ linear}{\_ flag}}} \right)*{lmcs\_ delta}{\_ abs}{\_ linear}{\_ cw}}} & \left\lbrack {{Equation}13} \right\rbrack\end{matrix}$ $\begin{matrix}{{{lmcs\_ cw}{\_ linear}} = {{{lmcs\_ cw}{\_ linear}{\_ delta}} + {OrgCW}}} & \left\lbrack {{Equation}14} \right\rbrack\end{matrix}$

In another example of this embodiment, when lmcs_cw_linear represents amapped value in lmcs_pivots_linear[lmcs_max_bin_idx+1] (i.e., P2 of FIG.10), lmcs_cw_linear_delta and lmcs_cw_linear may be derived based on thefollowing equations.

$\begin{matrix}{{{lmcs\_ cw}{\_ linear}{\_ delta}} = {\left( {1 - {2*{lmcs\_ delta}{\_ sign}{\_ cw}{\_ linear}{\_ flag}}} \right)*{lmcs\_ delta}{\_ abs}{\_ linear}{\_ cw}}} & \left\lbrack {{Equation}15} \right\rbrack\end{matrix}$ $\begin{matrix}{{{lmcs\_ cw}{\_ linear}} = {{{lmcs\_ cw}{\_ linear}{\_ delta}} + {{OrgCW}*\left( {{{lmcs\_ max}{\_ bin}{\_ idx}} - {{lmcs\_ max}{\_ bin}{\_ idx}} + 1} \right)}}} & \left\lbrack {{Equation}16} \right\rbrack\end{matrix}$

In the above-described equations, OrgCW may be a value derived based onTable 8 described above.

FIG. 11 illustrates one example of linear reshaping (or linearreshaping, linear mapping) according to an embodiment of the presentdocument. That is, in this embodiment, the use of a linear reshaper inLMCS is proposed. For example, this example in FIG. 11 may relate toforward linear reshaping (mapping).

In the existing example, the LMCS may use a piecewise linear mappingwith a fixed 16 pieces. Accordingly, since unavoidable deterioration mayoccur due to a sudden transition between pivot points, the complexity ofdesigning the reshaper may increase. Also, for inverse luma mapping ofreshaper, it requires to identify the piecewise function index. Thepiecewise function index identification process is an iteration processwith many comparisons. Furthermore, for chroma residual scaling usingthe corresponding luma block average, it also requires such lumapiecewise index identification process. This not only has the complexityissue, it also causes the latency of chroma residual scaling dependingon reconstruction of whole luma block. In order to solve this problem,the use of the linear reshaper is proposed for the LMCS.

Referring to FIG. 11, the linear reshaper may include two pivot pointsi.e., P1 and P2. P1 and P2 may represent input and mapped values, forexample P1 may be (min_input, 0) and P2 may be (max_input, max_mapped).Here, min_input represents the minimum input value, and max_inputrepresents the maximum input value. Any input value less than or equalto min_input are mapped to 0, any input value larger than max_input aremapped to max_mapped. Any input luma values within the min_input andmax_input are linearly mapped to other values. FIG. 11 shows an exampleof mapping. The pivot points P1, P2 may be determined at the encoder,and a linear fitting may be used to approximate the piecewise linearmapping.

There are many ways to signal the linear reshaper. In an example of amethod of signaling the linear reshaper, original luma range may bedivided by an equal number of bins. That is, the luma mapping betweenthe minimum and maximum bins may be equally distributed. For example,all bins may have same LMCS codeword (lmcsCW). To this end, minimum andmaximum bin indices may be signaled. Therefore, only one set ofreshape_model_bin_delta_abs_CW (or reshaper_model_delta_abs_CW,lmcs_delta_abs_CW) and reshaper_model_bin_delta_sign_CW_flag (orreshaper_model_delta_sign_CW_flag_sign_CW_flag) need to be signaled.

The following tables exemplarily show the syntax and semantics ofsignaling a linear reshaper according to this example.

TABLE 17 Descriptor lmcs_data ( ) {  log2_lmcs_num_bins_minus4 ue(v) lmcs_min_bin_idx ue(v)  lmcs_delta_max_bin_idx ue(v) lmcs_delta_cw_prec_minus1 ue(v)  lmcs_delta_abs_CW_linear u(v)  if (lmcs_delta_abs_CW_linear > 0 )   lmcs_delta_sign_CW_linear_flag u(1)

TABLE 18 log2_lmcs_num_bins_minus4 + 4 equals to the log2 of the numberof bins. log2_lmcs_num_bins = _log2_lmcs_num_bins_minus4 + 4 and is inthe range of 4 and BitDepth_(γ) lmcs_min_bin_idx specifies the minimumbin index used in the luma mapping with chroma scaling constructionprocess. The value of lmcs_min_bin_idx shall be in the range of 0 to 15,inclusive. lmcs_delta_max_bin_idx specifies the delta value between 15and the maximum bin index LmcsMaxBinIdx used in the luma mapping withchroma scaling construction process. The value of lmcs_delta_max_bin_idxshall be in the range of 0 to 15, inclusive. The value of LmcsMaxBinIdxis set equal to 15 − lmcs_delta_max_bin_idx. The value of LmcsMaxBinIdxshall be greater than or equal to lmcs_min_bin_idx.lmcs_delta_cw_prec_minus1 plus 1 specifies the number of bits used forthe representation of the syntax lmcs_delta_abs_cw[ i ]. The value oflmcs_delta_cw_prec_minus1 shall be in the range of 0 to BitDepthY − 2,inclusive. lmcs_delta_abs_cw_linear specifies the absolute deltacodeword value of one bin of linear mapping.lmcs_delta_sign_cw_linear_flag specifics the sign of the variablelmcsDeltaCWLinear as follows: - If lmcs_delta_sign_cw_linear_flag isequal to 0, lmcsDeltaCWLinear is a positive value. - Otherwise (lmcs_delta_sign_cw_linear_flag is not equal to 0 ), lmcsDeltaCWLinear isa negative value. When lmcs_delta_sign_cw_linear_flag is not present, itis inferred to be equal to 0. The variable OrgCW is derived as follows:OrgCW = 1 << (BitDepthY − log2_lmcs_num_bins) The variablelmcsDeltaCWLinear is derived as follows: lmcsDeltaCWLinear = ( 1 − 2 *lmcs_delta_sign_cw_linear_flag ) * lmcs_delta_abs_cw_linear lmcsCWLinear= OrgCW + lmcsDeltaCW_linear The variable ScaleCoeffSingle andInvScaleCoeffSingle are derived as follows: ScaleCoeffSingle = (lmcsCWLinear * (1 << FP_PREC) + (1 << (Log2(OrgCW) − 1))) >>(Log2(OrgCW)) InvScaleCoeffSingle = OrgCW * (1 << FP_PREC) /lmcsCWLinear The variable ChromaScaleCoeffSingle is derived as follows:ChromaScaleCoeffSingle = InvScaleCoeffSingle >> (FP_PREC−CSCALE_FP_PREC)

Referring to Table 17 and Table 18, the syntax element log2_lmcs_num_bins_minus4 may be information on the number of bins. Basedon this information, the number of bins may be signaled to allow finercontrol of the minimum and maximum pivot points. In another existingexample, the encoder and/or decoder may derive (specify) the (fixed)number of bins without signaling, for example the number of bins may bederived as 16 or 32. However, according to the examples of Table 17 andTable 18, log 2_lmcs_num_bins_minus4 plus 4 may represent a binarylogarithm of the number of bins. The number of bins derived based on thesyntax element may range from 4 to a value of the luma bit depthBitDepthY.

In Table 18, ScaleCoeffSingle may be referred to as a single lumaforward scaling factor, and InvScaleCoeffSingle may be referred to as asingle luma inverse scaling factor. (forward) mapping on predicted lumasamples may be performed based on a single luma forward scaling factor,and (inverse) mapping on reconstructed luma samples may be performedbased on a single luma inverse scaling factor. ChromaScaleCoeffSingle,as described above, may be referred to as a single chroma residualscaling factor. ScaleCoeffSingle, InvScaleCoeffSingle, andChromaSclaeCoeffSingle may be used for forward luma mapping, inverseluma mapping, and chroma residual scaling, respectively.ScaleCoeffSingle, InvScaleCoeffSingle, and ChromaSclaeCoeffSingle can beuniformly applied to all bins (16 bins PWL mappings) as a single factor.

Referring to Table 18, FP_PREC and CSCALE_FP_PREC may be constants forbit shifting. FP_PREC and CSCALE_FP_PREC may or may not be identical toeach other. For example, FP_PREC may be greater than or equal toCSCALE_FP_PREC. In one example, FP_PREC and CSCALE_FP_PREC may both be11. In another example, FP_PREC may be 15 and CSCALE_FP_PREC may be 11.

In another example of a method of signaling a linear reshaper, an LMCScodeword (lmcsCWlinearALL) may be derived based on the followingequations. In this example as well, linear reshaping syntax elementssignaled according to the syntax of Table 17 described above may beused. The following table shows an example of semantics describedaccording to this example.

TABLE 19 lmcs_delta_abs_cw_linear specifies the absolute delta codewordvalue between pivot points lmcs_min_bin_idx and lmcs_max_bin_idx +1. Thevariable lmcsDeltaCWLinear is derived as follows:  lmcsDeltaCWLinear = (1 − 2 * lmcs_delta_sign_cw_linear_flag ) * lmcs_delta_abs_cw_linear lmcsCWLinearAll = lmcsDeltaCWLinear + OrgCW * (lmcs_max_bin_idx−lmcs_min_bin_idx+1) lmcsCWLinearAll specifies the code words betweenlmcs_min_bin_idx and LmcsMaxBinIdx +1. The variable ScaleCoeffSingle andInvScaleCoeffSingle are derived as follows:  tmp = (LmcsCWLinearAll * (1<< FP PREC) ),/(lmcs_max_bin_idx lmcs_max_bin_idx+1);  ScaleCoeffSingle= (tmp + (1 << (Log2(OrgCW) − 1))) >> Log2(OrgCW)  InvScaleCoeffSingle =OrgCW*(lmcs_max_bin_idx lmcs_max_bin_idx+1 )*(1<<FP_PREC)   /lmcsCWLinearAll The variable ChromaScaleCoeffS ingle is derived asfollows:  ChromaScaleCoeffSingle = InvScaleCoeffSingle >> (FP_PREC−CSCALE_FP_PREC)

Referring to Table 19, FP_PREC and CSCALE_FP_PREC may be constants forbit shifting. FP_PREC and CSCALE_FP_PREC may or may not be identical toeach other. For example, FP_PREC may be greater than or equal toCSCALE_FP_PREC. In one example, FP_PREC and CSCALE_FP_PREC may both be11. In another example, FP_PREC may be 15 and CSCALE_FP_PREC may be 11.

In Table 19, lmcs_max_bin_idx may be used interchangeably withLmcsMaxBinIdx. lmcs_max_bin_idx and LmcsMaxBinIdx may be derived withreference to Table 18 described above based on lmcs_delta_max_bin_idx ofthe syntax of Table 17 described above. As such, Table 19 may beinterpreted with reference to Table 18.

In another embodiment according to this document, another example of amethod for signaling a linear reshaper may be proposed. The pivot pointsP1, P2 of the linear reshaper model may be explicitly signaled. Thefollowing tables show an example of syntax and semantics for explicitlysignaling a linear reshaper model according to this example.

TABLE 20 Descriptor lmcs_data ( ) {  lmcs_min_input ue(v) lmcs_max_input ue(v)  lmcs_max_mapped ue(v)

TABLE 21 lmcs_min_input specifies the input value of the 1st pivotpoint. It has a mapped value of 0. lmcs_max_input is the input value ofthe 2nd pivot point. lmcs_max_mapped is the mapped value at the 2^(nd)pivot points. They may be signaled explicitly using explo golomb code orfixed length code with length depending on BitDepth_(γ). lmcsCWLinearAll = lmcs_max_mapped The variable ScaleCoeffSingle andInvScaleCoeffSingle are derived as follows:  Rounding = (lmcs_max_input−lmcs_min_input)>>1  ScaleCoeffSingle = ( lmcsCWLinearAll * (1 <<FP_PREC) + Rounding))   / (lmcs_max_input−lmcs_min_input); InvScaleCoeffSingle = (lmcs_max_input− lmcs_min_input)*(1<<FP_PREC) /lmcsCWLinearAll The variable ChromaScaleCoeffSingle is derived asfollows: ChromaScaleCoeffSingle = InvScaleCoeffSingle >> (FP_PREC−CSCALE_FP_PREC)

Referring to Tables 20 and 21, the input value of the first pivot pointmay be derived based on the syntax element lmes_min_input, and the inputvalue of the second pivot point may be derived based on the syntaxelement lmes_max_input. The mapped value of the first pivot point may bea predetermined value (a value known to both the encoder and decoder),for example 0. A mapped value of the second pivot point may be derivedbased on the syntax element lmes_max_mapped. That is, the linearreshaper model may be explicitly (directly) signaled based on theinformation signaled based on the syntax of Table 20.

Alternatively, lmes_max_input and lmes_max_mapped may be signaled asdelta values. The following tables show an example of syntax andsemantics of signaling a linear reshaper model as a delta value.

TABLE 22 Descriptor lmcs_data ( ) {  lmcs_min_input ue(v) lmcs_max_input_delta ue(v)  lmcs_max_mapped_delta ue(v)

TABLE 23 lmcs_max_input_delta specifies the difference between the inputvalue of the 2nd pivot point to the max luma value (1<<bitdepthY)−1, and lmcs_max_input = (1<<bitdepthY)−1 − lmcs_max_input_delta;lmcs_max_mapped_delta specifies the difference between the mapped valueof the 2nd pivot point to the max luma value (1<<bitdepthY)−1. lmcsCWLinearAll = lmcs_max_mapped = (1<<bitdepthY)−1 −lmcs_max_mapped_delta The variable ScaleCoeffSingle andInvScaleCoeffSingle arc derived as follows:  Rounding = (lmcs_max_input−lmcs_min_input)>>1  ScaleCoeffSingle = ( lmcsCWLinearAll * (1 <<FP_PREC) + Rounding))   / (lmcs_max_input−lmcs_min_input); InvScaleCoeffSingle = (lmcs_max_input− lmcs_min_input)*(1<<FP_PREC) /lmcsCWLinearAll The variable ChromaScaleCoeffSingle is derived asfollows:  ChromaScaleCoeffSingle = InvScaleCoeffSingle >> (FP_PREC−CSCALE_FP_PREC)

Referring to Table 23, the input value of the first pivot point may bederived based on the syntax element lmcs_min_input. For example,lmcs_min_input may have a mapped value of 0. lmcs_max_input_delta mayrepresent a difference between the input value of the second pivot pointand the maximum luma value (i.e., (1<<bitdepthY)−1).lmcs_max_mapped_delta may indicate a difference between the mapped valueof the second pivot point and the maximum luma value (i.e.,(1<<bitdepthY)−1).

According to an embodiment of the present document, forward mapping forluma prediction samples, inverse mapping for luma reconstructionsamples, and chroma residual scaling may be performed based on theabove-described examples of the linear reshaper. In one example, onlyone inverse scaling factor may be needed for inverse scaling for luma(reconstructed) samples (pixels) in inverse mapping based linearreshaper. This is also true for forward mapping and chroma residualscaling. That is, the steps of determining ScaleCoeff[i],InvScaleCoeff[i] and ChromaScaleCoeff[i] for the bin index i may besubstituted for using only one factor. Here, one factor may refer to a(forward) slope or an inverse slope of the linear mapping expressed as afixed point. In one example, the inverse luma mapping scaling factor(inverse scaling factor in inverse mapping for luma reconstructionsamples) may be derived based on at least one of the followingequations.

$\begin{matrix}{{InvScaleCoeffSingle} = {{OrgCW}/{lmcsCWLinear}}} & \left\lbrack {{Equation}17} \right\rbrack\end{matrix}$ $\begin{matrix}{{{InvScaleCoeffSingle} = {{OrgCW}*\left( {{{lmcs\_ max}{\_ bin}{\_ idx}} - {{lmcs\_ max}{\_ bin}{\_ idx}} + 1} \right)/}}\text{ }{lmcsCWLinearAll}} & \left\lbrack {{Equation}18} \right\rbrack\end{matrix}$ $\begin{matrix}{{InvScaleCoeffSingle} = {\left( {{{lmcs\_ max}{\_ input}} - {{lmcs\_ min}{\_ input}}} \right)/{lmcsCWLinearAll}}} & \left\lbrack {{Equation}19} \right\rbrack\end{matrix}$

The lmcsCWLinear of Equation 17 may be derived from Tables 17 and 18described above. lmcsCWLinearALL of Equations 18 and 19 may be derivedfrom at least one of Tables 19 to 23 described above. In Equation 17 or18, OrgCW may be derived from Table 8 or Table 18.

The following tables describe equations and syntax (conditionalstatements) indicating a forward mapping procedure for luma samples(i.e., luma prediction samples) in picture reconstruction. In thefollowing tables and equations, FP_PREC is a constant for bit shifting,and may be a predetermined value. For example, FP_PREC may be 11 or 15.

TABLE 24 idxY = predSamples└ i ┘└ j ┘ >> Log2( OrgCW ) PredMapSamples[ i][ j ] = LmcsPivot[ idxY ]   + ( ScaleCoeff[ idxY ] * ( predSamples[ i][ j ] − InputPivot[ idxY ] ) + ( 1 << 10 ) ) >> 11  with i = 0..nCurrSw− 1, j = 0..nCurrSh − 1

TABLE 25 if (PredMapSamples[ i ][ j ] <= lmcs_min_input) PredMapSamples[ i ][ j ]=0 else if (PredMapSamples[ i ][ j ] >=lmcs_max_input)  PredMapSamples[ i ][ j ]=lmcs_max_mapped else PredMapSamples[ i ][ j ] = (ScaleCoeffSingle * predSamples[ i ][ j ]  + ( 1<<( FP_PREC−1 ) ) )>> FP_PREC

Table 24 may be for deriving forward mapped luma samples in the lumamapping procedure based on Tables 7 and 8 described above. That is,Table 24 may be described together with Tables 7 and 9. In Table 24, theforward mapped luma (prediction) samples PredMapSamples[i][j] as outputcan be derived from luma (prediction) samples predSamples[i][j] asinput. idxY of Table 24 may be referred to as a (forward) mapping index,and the mapping index may be derived based on predicted luma samples.

Table 25 may be for deriving forward mapped luma samples in luma mappingaccording to application of the linear reshaper. For example,lmcs_min_input, lmcs_max_input, lmcs_max_mapped, and ScaleCoeffSingle ofTable 25 may be derived by at least one of Tables 20 to 23. In Table 25,forward mapped luma (prediction) samples PredMapSamples[i][j] can bederived as output from luma (prediction) samples as input predSamples[i][U] when lmcs_min_input<predSamples[i][j]<lmcs_max_input. Throughcomparison between Table 24 and Table 25, the change from the existingLMCS according to the application of the linear reshaper can be seenfrom the perspective of forward mapping.

The following equations describe an inverse mapping procedure for lumasamples (i.e., luma reconstruction samples). In the following equations,a lumaSample as an input may be a luma reconstruction sample beforeinverse mapping (before modification). The invSample as output may be aninverse mapped (modified) luma reconstruction sample. In other cases,the clipped invSample may be referred to as a modified lumareconstruction sample.

$\begin{matrix}{\left. {{invSample} = {{{InputPivot}\lbrack{idxYInv}\rbrack} + \left( {{{{InvScaleCoeff}\lbrack{idxYInv}\rbrack}*\left( {{lumaSample} - {{LmcsPivot}\lbrack{idxYInv}\rbrack}} \right)} + \left( {1{\operatorname{<<}\text{FP\_PREC-1}}} \right)} \right)}} \right)\operatorname{>>}{FP\_ PREC}} & \left\lbrack {{Equation}20} \right\rbrack\end{matrix}$ $\begin{matrix}{\left. {{invSample} = {{{lmcs\_ min}{\_ input}} + \left( {{{InvScaleCoeffSingle}*\left( {{lumaSample} - {{lmcs\_ min}{\_ input}}} \right)} + \left( {1{\operatorname{<<}\text{FP\_PREC-1}}} \right)} \right)}} \right)\operatorname{>>}{FP\_ PREC}} & \left\lbrack {{Equation}21} \right\rbrack\end{matrix}$

Referring to Equation 20, the index idxInv maybe derived based on Table10 described above. That is, Equation 20 may be for deriving inversemapped luma samples in the luma mapping procedure based on Tables 7 and8 described above. Equation 20 can be described together with Table 9described above.

Equation 21 may be for deriving inverse mapped luma samples in lumamapping according to the application of the linear reshaper. Forexample, lmcs_min_input of Equation 21 may be derived according to atleast one of Tables 20 to 23. Through the comparison between Equation 20and Equation 21, the change from the existing LMCS according to theapplication of the linear reshaper can be seen from the perspective offorward mapping.

Based on the above-described examples of the linear reshaper, thepiecewise index identification process may be omitted. That is, in thepresent examples, since there is only one piece having valid reshapedluma pixels, the piecewise index identification process used for inverseluma mapping and chroma residual scaling can be removed. Accordingly,the complexity of inverse luma mapping may be reduced. In addition,latency problems caused by depending on luma piecewise indexidentification during chroma residual scaling can be eliminated.

According to the embodiment of the use of the linear reshaper describedabove, the following advantages may be provided for LMCS: i) It ispossible to simplify the encoder reshaper design, preventing possibleartifact by abrupt changes between the piecewise linear pieces ii) Thedecoder inverse mapping process, in which the piecewise indexidentification process can be removed, can be simplified by eliminatingthe piecewise index identification process iii) By removing thepiecewise index identification process, it is possible to remove thelatency problem in the chroma residual scaling caused by depending onthe corresponding luma blocks iv) It is possible to reduce overhead ofsignaling, and make frequent update of reshaper more feasible v) Formany places that used to require of a loop of 16 pieces, the loop can beeliminated. For example, to derive InvScaleCoeff[i], the number ofdivision operations by lmcsCW[i] can be reduced to 1.

In another embodiment according to this document, an LMCS based onflexible bins is proposed. Here, flexible bins may mean that the numberof bins is not fixed to a predetermined number. In an existingembodiment, the number of bins in the LMCS is fixed to 16, and the 16bins are evenly distributed for input sample values. In this embodiment,a flexible number of bins is proposed and the pieces (bins) will not beevenly distributed with respect to the original pixel values.

The following tables exemplarily show syntax and/or semantics regardingLMCS data according to the present embodiment.

TABLE 26 Descriptor lmcs_data ( ) {  lmcs_num_bins_minus1 ue(v)  for ( i= 0; i <= lmcs_num_bins; i++ ) { u(v)   lmcs_delta_input_cw[ i ] u(v)  lmcs_delta_mapped_cw[ i ] u(v)  }

TABLE 27 lmcs_num_bins_minus1 plus 1 specifies the number of bins.lmcs_num_bins shall be in the range of 1 and 1<<BitDepth_(γ)−1.lmcs_num_bins or lmcs_num_bins_minus1 may be restricted to multiple ofpower of 2, or log of 2 to reduce the number of bits used for signaling.lmcs_num_bins = lmcs_num_bins_minus1 + 1 lmcs_delta_input_cw[ i ]specifies the delta input value of the ith pivot point relative theprevious pivot point. lmcs_delta_input_cw[ i ] >= 0.lmcs_delta_mapped_cw[ i ] specifies the delta mapped value of the ithpivot point relative the previous pivot point. lmcs_delta_mapped_cw[ i] >= 0 The variable LmcsPivot_input[ i ] and LmcsPivot_mapped[ i ] withi = 0..lmcs_num_bins+1 LmcsPivot_input[ 0 ] = 0; LmcsPivot_mapped[ 0 ] =0; for( i = 0; i <= lmcs_num_bins; i++ ) {  LmcsPivot_input[ i + 1 ] =LmcsPivot_input[ i ] + lmcs_delta_input_ cw[ i ];   LmcsPivot_mapped[i + 1 ] = LmcsPivot_mapped[ i ] + lmcs_delta_mapped_cw[ i ]; }LmcsPivot_input[ lmcs_num_bins+1] shall equal to (1 << BitDepth_(Y) ) −1 . I.e. sum of all lmcs_delta_input_cw[ i ] shall equal to (1 <<BitDepth_(γ) ) − 1, therefore the last lmcs_delta_mapped_cw[ i ] can beinferred without signaling. LmcsPivot_mapped└ lmcs_num_bins+1┘, I.e. sumof all lmcs_delta_mapped_cw└ i ┘ shall be not greater than (1 <<BitDepth_(γ) ) − 1.

Referring to Table 26, information on the number of binslmcs_num_bins_minus1 may be signaled. Referring to Table 27,lmcs_num_bins_minus1+1 may be equal to the number of bins, and fromthis, the number of bins may be in a range from 1 to (1<<BitDepthY)−1.For example, lmcs_num_bins_minus1 or lmcs_num_bins_minus1+1 may be amultiple of 2.

In the embodiment described together with tables 26 and 27, the numberof pivot points can, regardless of whether the reshaper is linear, bederived based on lmcs_num_bins_minus1 (information on the number ofbins) (signaling of lmcs_num_bins_minus1) and input values and mappedvalues of pivot points (LmcsPivot_input[i], LmcsPivot_mapped[i]) may bederived based on the summation of signaled codeword values(lmcs_delta_input_cw[i], lmcs_delta_mapped_cw[i]) (here, the initialinput value LmcsPivot_input[0] and the initial output valueLmcsPivot_mapped[0] are 0).

The following drawings are created to explain specific examples of thepresent specification. Since the names of specific devices described inthe drawings or the names of specific signals/messages/fields arepresented by way of example, the technical features of the presentspecification are not limited to the specific names used in thefollowing drawings.

FIG. 12 and FIG. 13 schematically show an example of a video/imageencoding method and related components according to embodiment(s) of thepresent document. The method disclosed in FIG. 12 may be performed bythe encoding apparatus disclosed in FIG. 2. Specifically, for example,S1200 may be performed by the residual processor 230 or the predictor220 of the encoding apparatus, S1210 may be performed by the predictor220 or the adder 250 of the encoding apparatus, S1220 may be performedby the residual processor 230 or the adder 250 of the encodingapparatus, S1230, S1240 or S1250 may be performed by the residualprocessor 230 of the encoding apparatus, and S1260 may be performed bythe entropy encoder 240 of the encoding apparatus. The method disclosedin FIG. 12 may include the embodiments described above in the presentdocument.

Referring to FIG. 12, the encoding apparatus may generate predictionluma samples and residual chroma samples (S1200). With respect to theprediction luma samples, the encoding apparatus may derive theprediction luma samples of the current block based on the predictionmode. In this case, various prediction methods disclosed in thisdocument, such as inter prediction or intra prediction, may be applied.Similarly, the encoding apparatus may derive prediction chroma samples.The encoding apparatus may derive the residual chroma samples based onthe original chroma samples and the prediction chroma samples of thecurrent block. For example, the encoding apparatus may derive residualchroma samples based on a difference between the prediction chromasamples and the original chroma samples.

The encoding apparatus may generate mapped prediction luma samples(S1210). For example, the encoding apparatus may derive input values andmapping values (output values) of pivot points for luma mapping, and maygenerate mapped prediction luma samples based on the input values andmapping values. In an example, the encoding apparatus may derive amapping index (idxY) based on the first prediction luma sample, and maygenerate the first mapped prediction luma samples based on the inputvalue and the mapping value of the pivot point corresponding to themapping index. In another example, linear mapping (linear reshaping,linear LMCS) may be used and mapped prediction luma samples may begenerated based on a forward mapping scaling factor derived from twopivot points in the linear mapping, thus The index derivation proceduremay be omitted by linear mapping.

The encoding apparatus may generate scaled residual chroma samples(S1220). Specifically, the encoding apparatus may derive a chromaresidual scaling factor and generate scaled residual chroma samplesbased on the chroma residual scaling factor. Here, the chroma residualscaling of the encoding side may be referred to as forward chromaresidual scaling. Accordingly, the chroma residual scaling factorderived by the encoding apparatus may be referred to as a forward chromaresidual scaling factor, and forward scaled residual chroma samples maybe generated.

The encoding apparatus may derive LMCS related information based on themapped predicted luma samples and the scaled residual chroma samples(S1230). The encoding apparatus may generate LMCS related informationfor the reconstructed samples. The encoding apparatus may derive LMCSrelated parameters that can be applied for filtering the reconstructedsamples, and may generate LMCS related information based on the LMCSrelated parameters. For example, the LMCS related information mayinclude information on the luma mapping (i.e., forward mapping, inversemapping, linear mapping), information on chroma residual scaling, and/orindices (i.e., a maximum bin index, a minimum bin index) related to LMCS(or reshaping, reshaper).

The encoding apparatus may generate residual luma samples based on themapped predicted luma samples (S1240). For example, the encodingapparatus may derive residual luma samples based on a difference betweenthe mapped predicted luma samples and original luma samples.

The encoding apparatus may derive residual information based on thescaled residual chroma samples and the residual luma samples (S1250).The encoding apparatus may derive transform coefficients based on atransform procedure for the scaled residual chroma samples and the lumaresidual samples. For example, the transform process may include atleast one of DCT, DST, GBT, or CNT. The encoding apparatus may derivequantized transform coefficients based on the quantization process forthe transform coefficients. The quantized transform coefficients mayhave a one-dimensional vector form based on the coefficient scan order.The encoding apparatus may generate residual information specifying thequantized transform coefficients. The residual information may begenerated through various encoding methods such as exponential Golomb,CAVLC, CABAC, and the like.

The encoding apparatus may encode the image/video information (S1260).The image information may include LMCS related information and/orresidual information. For example, the LMCS related information mayinclude information on the linear mapping. In one example, at least oneLMCS codeword may be derived based on information on the linear mapping.The encoded video/image information may be output in the form of abitstream. The bitstream may be transmitted to the decoding devicethrough a network or a storage medium.

The image/video information may include various information according toan embodiment of the present document. For example, the image/videoinformation may include information disclosed in at least one of Tables1, 3, 4, 7, 14, 15, 16, 17, 20, 22 or 26 described above.

In an embodiment, the LMCS related information may include informationon a linear LMCS, and the mapped predicted luma samples may be generatedbased on the information on the linear LMCS.

In one embodiment, generating the scaled residual chroma samplescomprises deriving a single chroma residual scaling factor, andgenerating the scaled residual chroma samples based on the residualchroma samples and the single chroma residual scaling factor. In oneexample, the image information may include information on the singlechroma residual scaling factor, and a value of the information on thesingle chroma residual scaling factor may be the same as a value of thesingle chroma residual scaling factor. In another example, the singlechroma residual scaling factor may be derived based on at least one ofEquations included in Tables 12, 13, 18, 19, 21, 23, or Equations 5, 6,9, 10, 11 or 12 described above. Through the derivation of reconstructedchroma samples based on a single chroma residual scaling factor, scalingin a chroma block may not depend on a luma block value so that piecewiseindex identification may not be required. Therefore, the chroma residualscaling procedure in the LMCS can be performed without a latency issue.

In an embodiment, the information on the linear mapping may includeinformation on a first pivot point (i.e., P1 in FIG. 11) and informationon a second pivot point (i.e., P2 in FIG. 11). For example, the inputvalue and the mapping value of the first pivot point may be a minimuminput value and a minimum mapping value, respectively. The input valueand the mapping value of the second pivot point may be a maximum inputvalue and a maximum mapping value, respectively. An input value betweenthe minimum input value and the maximum input value may be linearlymapped.

In an embodiment, the image information may include information on themaximum input value and information on the maximum mapping value. Themaximum input value may be the same as a value of information on themaximum input value (i.e., lmcs_max_input in Table 20). The maximummapping value may be the same as a value of information on the maximummapping value (i.e., lmcs_max_mapped in Table 20).

In an embodiment, the information on the linear mapping includesinformation on an input delta value of the second pivot point (i.e.,lmcs_max_input_delta in Table 22) and information on a mapping deltavalue of the second pivot point (i.e., lmcs_max_mapped_delta of Table22). The maximum input value may be derived based on the input deltavalue of the second pivot point, and the maximum mapping value may bederived based on the mapping delta value of the second pivot point.

In an embodiment, the maximum input value and the maximum mapping valuemay be derived based on at least one equation included in Table 23described above.

In an embodiment, generating the mapped prediction luma samples includesderiving a forward mapping scaling factor (i.e., ScaleCoeffSingle) forthe prediction luma samples, and generating the mapped prediction lumasamples based on the forward mapping scaling factor. The forward mappingscaling factor may be a single factor for the prediction luma samples.

In an embodiment, the forward mapping scaling factor may be derivedbased on at least one equation included in Tables 21 and/or 23 describedabove.

In an embodiment, the mapped prediction luma samples may be derivedbased on at least one equation included in Table 25 described above.

In an embodiment, the encoding apparatus may derive an inverse mappingscaling factor (i.e., InvScaleCoeffSingle) for the reconstructed lumasamples (i.e., the aforementioned lumaSample). Also, the encodingapparatus may generate modified reconstructed luma samples (i.e.,invSample) based on the reconstructed luma samples and the inversemapping scaling factor. The inverse mapping scaling factor may be asingle factor for the reconstructed luma samples.

In an embodiment, the inverse mapping scaling factor may be derivedbased on at least one Equation included in Tables 18, 19, 21, and 23, orEquation 11 or 12 described above.

In an embodiment, the modified reconstructed luma samples may be derivedbased on Equation 21 described above.

In an embodiment, the LMCS related information may include informationon the number of bins for deriving the mapped prediction luma samples(i.e., lmcs_num_bins_minus1 in Table 26). For example, the number ofpivot points for luma mapping may be set equal to the number of bins. Inone example, the encoding apparatus may generate the delta input valuesand delta mapping values of the pivot points by the number of the bins,respectively. In one example, the input values and mapping values of thepivot points are derived based on the delta input values (i.e.,lmcs_delta_input_cw[i] in Table 26) and the delta mapping values (i.e.,lmcs_delta_mapped_cw[i] in Table 26), and the mapped prediction lumasamples may be generated based on the input values (i.e.,LmcsPivot_input[i] of Table 27, or InputPivot[i] of Table 8) and themapping values (i.e., LmcsPivot_mapped[i] of Table 27, or LmcsPivot[i]of Table 8).

In an embodiment, the encoding apparatus may derive an LMCS deltacodeword based on at least one LMCS codeword and an original codeword(OrgCW) included in the LMCS related information, and mapped lumaprediction samples may be derived based on the at least one LMCScodeword and the original code. In one example, the information on thelinear mapping may include information on the LMCS delta codeword.

In one embodiment, the at least one LMCS codeword may be derived basedon the summation of the LMCS delta codeword and OrgCW, for example,OrgCW is (1<<BitDepthY)/16, where BitDepthY is a luma bit depth. Thisembodiment may be based on Equation 12.

In an embodiment, the at least one LMCS codeword may be derived based onthe summation of the LMCS delta codeword andOrgCW*(lmcs_max_bin_idx-lmcs_min_bin_idx+1), and lmcs_max_bin_idx andlmcs_min_bin_idx are respectively a maximum bin index and a minimum binindex., and OrgCW may be (1<<BitDepthY)/16. This embodiment may be basedon Equations 15 and 16.

In one embodiment, the at least one LMCS codeword may be a multiple oftwo.

In one embodiment, when the luma bit depth (BitDepthY) of thereconstructed luma samples is higher than 10, the at least one LMCScodeword may be a multiple of 1<<(BitDepthY−10).

In one embodiment, the at least one LMCS codeword may be in the rangefrom (OrgCW>>1) to (OrgCW<<1)−1.

FIG. 14 and FIG. 15 schematically show an example of an image/videodecoding method and related components according to an embodiment of thepresent document. The method disclosed in FIG. 14 may be performed bythe decoding apparatus illustrated in FIG. 3. Specifically, for example,S1400 of FIG. 14 may be performed by the entropy decoder 310 of thedecoding apparatus, S1410 may be performed by the predictor 330 of thedecoding apparatus, S1420 may be performed by the residual processor 320of the decoding apparatus, S1430 may be performed by the residualprocessor 320, the predictor 330 and/or the adder 340 of the decodingapparatus, S1440 may be performed by the adder 340 of the decodingapparatus, S1450 may be performed by the residual processor 320 or theadder 340 of the decoding apparatus, and S1460 may be performed by theadder 340 of the decoding apparatus. The method disclosed in FIG. 14 mayinclude the embodiments described above in the present document.

Referring to FIG. 14, the decoding apparatus may receive/obtainvideo/image information (S1400). The video/image information may includeLMCS related information. For example, the LMCS related information mayinclude information on the luma mapping (i.e., forward mapping, inversemapping, linear mapping), information on chroma residual scaling, and/orindices (i.e., a maximum bin index, a minimum bin index, a mappingindex) related to LMCS (or reshaping, reshaper). The decoding apparatusmay receive/obtain the image/video information through a bitstream.

The image/video information may include various information according toan embodiment of the present document. For example, the image/videoinformation may include information disclosed in at least one of Tables1, 3, 4, 7, 14, 15, 16, 17, 20, 22 and/or 26 described above.

The decoding apparatus may generate prediction luma samples (S1410). Thedecoding apparatus may derive the prediction luma samples of the currentblock based on the prediction mode. In this case, various predictionmethods disclosed in this document, such as inter prediction or intraprediction, may be applied.

The decoding apparatus may generate residual chroma samples based on theresidual information (S1420). Specifically, the decoding apparatus mayderive quantized transform coefficients based on the residualinformation. The quantized transform coefficients may have aone-dimensional vector form based on a coefficient scan order. Thedecoding apparatus may derive transform coefficients based on an inversequantization procedure for the quantized transform coefficients. Thedecoding apparatus may derive residual chroma samples and/or residualluma samples based on the transform coefficients.

The decoding apparatus may generate mapped prediction luma samples(S1430). For example, the decoding apparatus may derive input values andmapping values (output values) of pivot points for luma mapping, and maygenerate mapped prediction luma samples based on the input values andmapping values. In one example, the decoding apparatus may derive a(forward) mapping index (idxY) based on the first prediction lumasample, and may generate the first mapped prediction luma samples basedon the input value and the mapping value of the pivot pointcorresponding to the mapping index. In another example, linear mapping(linear reshaping, linear LMCS) may be used and mapped prediction lumasamples may be generated based on a forward mapping scaling factorderived from two pivot points in the linear mapping, thus the indexderivation procedure may be omitted by linear mapping.

The decoding apparatus may generate reconstructed luma samples (S1440).The decoding apparatus may generate reconstructed luma samples based onthe mapped prediction luma samples. Specifically, the decoding apparatusmay sum the above-described residual luma samples with the mappedprediction luma samples, and may generate reconstructed luma samplesbased on a result of the summation.

The decoding apparatus may generate scaled residual chroma samples(S1450). Specifically, the decoding apparatus may derive a chromaresidual scaling factor and generate scaled residual chroma samplesbased on the chroma residual scaling factor. Here, the chroma residualscaling of the decoding side may be referred to as inverse chromaresidual scaling, as opposed to the encoding side. Accordingly, thechroma residual scaling factor derived by the decoding apparatus may bereferred to as an inverse chroma residual scaling factor, and inversescaled residual chroma samples may be generated.

The decoding apparatus may generate reconstructed chroma samples(S1460). The decoding apparatus may generate reconstructed chromasamples based on the scaled residual chroma samples. Specifically, thedecoding apparatus may perform a prediction procedure on the chromacomponent and generate prediction chroma samples. The decoding apparatusmay generate reconstructed chroma samples based on the summation betweenthe prediction chroma samples and the scaled residual chroma samples.

In one embodiment, the LMCS related information may include informationon a linear LMCS, and the mapped predicted luma samples may be generatedbased on the information on the linear LMCS.

In one embodiment, generating the scaled residual chroma samplescomprises deriving a single chroma residual scaling factor, andgenerating the scaled residual chroma samples based on the residualchroma samples and the single chroma residual scaling factor. In oneexample, the image information may include information on the singlechroma residual scaling factor, and a value of the information on thesingle chroma residual scaling factor may be the same as a value of thesingle chroma residual scaling factor. In another example, the singlechroma residual scaling factor may be derived based on at least one ofEquations included in Tables 12, 13, 18, 19, 21, 23, or Equations 5, 6,9, 10, 11 or 12 described above. Through the derivation of reconstructedchroma samples based on a single chroma residual scaling factor, scalingin a chroma block may not depend on a luma block value so that piecewiseindex identification may not be required. Therefore, the chroma residualscaling procedure in the LMCS can be performed without a latency issue.

In an embodiment, the information on the linear mapping may includeinformation on a first pivot point (i.e., P1 in FIG. 11) and informationon a second pivot point (i.e., P2 in FIG. 11). For example, the inputvalue and the mapping value of the first pivot point may be a minimuminput value and a minimum mapping value, respectively. The input valueand the mapping value of the second pivot point may be a maximum inputvalue and a maximum mapping value, respectively. An input value betweenthe minimum input value and the maximum input value may be linearlymapped.

In an embodiment, the image information may include information on themaximum input value and information on the maximum mapping value. Themaximum input value may be the same as a value of information on themaximum input value (i.e., lmcs_max_input in Table 20). The maximummapping value may be the same as a value of information on the maximummapping value (i.e., lmcs_max_mapped in Table 20).

In an embodiment, the information on the linear mapping includesinformation on an input delta value of the second pivot point (i.e.,lmcs_max_input_delta in Table 22) and information on a mapping deltavalue of the second pivot point (i.e., lmcs_max_mapped_delta of Table22). The maximum input value may be derived based on the input deltavalue of the second pivot point, and the maximum mapping value may bederived based on the mapping delta value of the second pivot point.

In an embodiment, the maximum input value and the maximum mapping valuemay be derived based on at least one equation included in Table 23described above.

In an embodiment, generating the mapped prediction luma samples includesderiving a forward mapping scaling factor (i.e., ScaleCoeffSingle) forthe prediction luma samples, and generating the mapped prediction lumasamples based on the forward mapping scaling factor. The forward mappingscaling factor may be a single factor for the prediction luma samples.

In an embodiment, the forward mapping scaling factor may be derivedbased on at least one equation included in Tables 21 and/or 23 describedabove.

In an embodiment, the mapped prediction luma samples may be derivedbased on at least one equation included in Table 25 described above.

In an embodiment, the decoding apparatus may derive an inverse mappingscaling factor (i.e., InvScaleCoeffSingle) for the reconstructed lumasamples (i.e., the aforementioned lumaSample). Also, the decodingapparatus may generate modified reconstructed luma samples (i.e.,invSample) based on the reconstructed luma samples and the inversemapping scaling factor. The inverse mapping scaling factor may be asingle factor for the reconstructed luma samples.

In an embodiment, the inverse mapping scaling factor may be derivedbased on at least one Equation included in Tables 18, 19, 21, and 23, orEquation 11 or 12 described above.

In an embodiment, the modified reconstructed luma samples may be derivedbased on Equation 21 described above.

In an embodiment, the LMCS related information may include informationon the number of bins for deriving the mapped prediction luma samples(i.e., lmcs_num_bins_minus1 in Table 26). For example, the number ofpivot points for luma mapping may be set equal to the number of bins. Inone example, the decoding apparatus may signal the delta input valuesand delta mapping values of the pivot points by the number of the bins,respectively. In one example, the input values and mapping values of thepivot points are derived based on the delta input values (i.e.,lmcs_delta_input_cw[i] in Table 26) and the delta mapping values (i.e.,lmcs_delta_mapped_cw[i] in Table 26), and the mapped prediction lumasamples may be generated based on the input values (i.e.,LmcsPivot_input[i] of Table 27, or InputPivot[i] of Table 8) and themapping values (i.e., LmcsPivot_mapped[i] of Table 27, or LmcsPivot[i]of Table 8).

In an embodiment, the decoding apparatus may derive an LMCS deltacodeword based on at least one LMCS codeword and an original codeword(OrgCW) included in the LMCS related information, and mapped lumaprediction samples may be derived based on the at least one LMCScodeword and the original code. In one example, the information on thelinear mapping may include information on the LMCS delta codeword.

In one embodiment, the at least one LMCS codeword may be derived basedon the summation of the LMCS delta codeword and OrgCW, for example,OrgCW is (1<<BitDepthY)/16, where BitDepthY is a luma bit depth. Thisembodiment may be based on Equation 12.

In an embodiment, the at least one LMCS codeword may be derived based onthe summation of the LMCS delta codeword andOrgCW*(lmcs_max_bin_idx-lmcs_min_bin_idx+1), and lmcs_max_bin_idx andlmcs_min_bin_idx are respectively a maximum bin index and a minimum binindex., and OrgCW may be (1<<BitDepthY)/16. This embodiment may be basedon Equations 15 and 16.

In one embodiment, the at least one LMCS codeword may be a multiple oftwo.

In one embodiment, when the luma bit depth (BitDepthY) of thereconstructed luma samples is higher than 10, the at least one LMCScodeword may be a multiple of 1<<(BitDepthY−10).

In one embodiment, the at least one LMCS codeword may be in the rangefrom (OrgCW>>1) to (OrgCW<<1)−1.

In the above-described embodiment, the methods are described based onthe flowchart having a series of steps or blocks. The present disclosureis not limited to the order of the above steps or blocks. Some steps orblocks may occur simultaneously or in a different order from other stepsor blocks as described above. Further, those skilled in the art willunderstand that the steps shown in the above flowchart are notexclusive, that further steps may be included, or that one or more stepsin the flowchart may be deleted without affecting the scope of thepresent disclosure.

The method according to the above-described embodiments of the presentdocument may be implemented in software form, and the encoding deviceand/or decoding device according to the present document is, forexample, may be included in the device that performs the imageprocessing of a TV, a computer, a smart phone, a set-top box, a displaydevice, etc.

When the embodiments in the present document are implemented insoftware, the above-described method may be implemented as a module(process, function, etc.) that performs the above-described function. Amodule may be stored in a memory and executed by a processor. The memorymay be internal or external to the processor, and may be coupled to theprocessor by various well-known means. The processor may include anapplication-specific integrated circuit (ASIC), other chipsets, logiccircuits, and/or data processing devices. Memory may include read-onlymemory (ROM), random access memory (RAM), flash memory, memory cards,storage media, and/or other storage devices. That is, the embodimentsdescribed in the present document may be implemented and performed on aprocessor, a microprocessor, a controller, or a chip. For example, thefunctional units shown in each figure may be implemented and performedon a computer, a processor, a microprocessor, a controller, or a chip.In this case, information on instructions or an algorithm forimplementation may be stored in a digital storage medium.

In addition, the decoding apparatus and the encoding apparatus to whichthe present disclosure is applied may be included in a multimediabroadcasting transmission/reception apparatus, a mobile communicationterminal, a home cinema video apparatus, a digital cinema videoapparatus, a surveillance camera, a video chatting apparatus, areal-time communication apparatus such as video communication, a mobilestreaming apparatus, a storage medium, a camcorder, a VoD serviceproviding apparatus, an Over the top (OTT) video apparatus, an Internetstreaming service providing apparatus, a three-dimensional (3D) videoapparatus, a teleconference video apparatus, a transportation userequipment (i.e., vehicle user equipment, an airplane user equipment, aship user equipment, etc.) and a medical video apparatus and may be usedto process video signals and data signals. For example, the Over the top(OTT) video apparatus may include a game console, a blue-ray player, aninternet access TV, a home theater system, a smart phone, a tablet PC, aDigital Video Recorder (DVR), and the like.

Furthermore, the processing method to which the present document isapplied may be produced in the form of a program that is to be executedby a computer and may be stored in a computer-readable recording medium.Multimedia data having a data structure according to the presentdisclosure may also be stored in computer-readable recording media. Thecomputer-readable recording media include all types of storage devicesin which data readable by a computer system is stored. Thecomputer-readable recording media may include a BD, a Universal SerialBus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, a magnetic tape, afloppy disk, and an optical data storage device, for example.Furthermore, the computer-readable recording media includes mediaimplemented in the form of carrier waves (i.e., transmission through theInternet). In addition, a bitstream generated by the encoding method maybe stored in a computer-readable recording medium or may be transmittedover wired/wireless communication networks.

In addition, the embodiments of the present document may be implementedwith a computer program product according to program codes, and theprogram codes may be performed in a computer by the embodiments of thepresent document. The program codes may be stored on a carrier which isreadable by a computer.

FIG. 16 shows an example of a content streaming system to whichembodiments disclosed in the present document may be applied.

Referring to FIG. 16, the content streaming system to which theembodiment(s) of the present document is applied may largely include anencoding server, a streaming server, a web server, a media storage, auser device, and a multimedia input device.

The encoding server compresses content input from multimedia inputdevices such as a smartphone, a camera, a camcorder, etc. Into digitaldata to generate a bitstream and transmit the bitstream to the streamingserver. As another example, when the multimedia input devices such assmartphones, cameras, camcorders, etc. directly generate a bitstream,the encoding server may be omitted.

The bitstream may be generated by an encoding method or a bitstreamgenerating method to which the embodiment(s) of the present disclosureis applied, and the streaming server may temporarily store the bitstreamin the process of transmitting or receiving the bitstream.

The streaming server transmits the multimedia data to the user devicebased on a user's request through the web server, and the web serverserves as a medium for informing the user of a service. When the userrequests a desired service from the web server, the web server deliversit to a streaming server, and the streaming server transmits multimediadata to the user. In this case, the content streaming system may includea separate control server. In this case, the control server serves tocontrol a command/response between devices in the content streamingsystem.

The streaming server may receive content from a media storage and/or anencoding server. For example, when the content is received from theencoding server, the content may be received in real time. In this case,in order to provide a smooth streaming service, the streaming server maystore the bitstream for a predetermined time.

Examples of the user device may include a mobile phone, a smartphone, alaptop computer, a digital broadcasting terminal, a personal digitalassistant (PDA), a portable multimedia player (PMP), navigation, a slatePC, tablet PCs, ultrabooks, wearable devices (ex. Smartwatches, smartglasses, head mounted displays), digital TVs, desktops computer, digitalsignage, and the like. Each server in the content streaming system maybe operated as a distributed server, in which case data received fromeach server may be distributed.

Each server in the content streaming system may be operated as adistributed server, and in this case, data received from each server maybe distributed and processed.

The claims described herein may be combined in various ways. Forexample, the technical features of the method claims of the presentdocument may be combined and implemented as an apparatus, and thetechnical features of the apparatus claims of the present document maybe combined and implemented as a method. In addition, the technicalfeatures of the method claim of the present document and the technicalfeatures of the apparatus claim may be combined to be implemented as anapparatus, and the technical features of the method claim of the presentdocument and the technical features of the apparatus claim may becombined and implemented as a method.

What is claimed is:
 1. An image decoding method performed by a decodingapparatus, the method comprising: obtaining image information includingprediction related information, residual information, and luma mappingwith chroma scaling (LMCS) related information from a bitstream;generating prediction luma samples based on the prediction relatedinformation; generating residual chroma samples based on the residualinformation; generating mapped prediction luma samples based on the LMCSrelated information and the prediction luma samples; generatingreconstructed luma samples based on the mapped prediction luma samples;generating scaled residual chroma samples based on the LMCS relatedinformation and the residual chroma samples; and generatingreconstructed chroma samples based on the scaled residual chromasamples, wherein the LMCS related information includes information on alinear LMCS, and wherein the mapped prediction luma samples aregenerated based on the information on the linear LMCS.
 2. The method ofclaim 1, wherein generating the scaled residual chroma samplescomprises: deriving a single chroma residual scaling factor; andgenerating the scaled residual chroma samples based on the residualchroma samples and the single chroma residual scaling factor; whereinthe image information includes information on the single chroma residualscaling factor, and wherein a value of the information on the singlechroma residual scaling factor is equal to a value of the single chromaresidual scaling factor.
 3. The method of claim 1, wherein theinformation one the linear mapping includes information one a firstpivot point and information on a second pivot point, wherein an inputvalue and a mapping value of the first pivot point are a minimum inputvalue and a minimum mapping value, respectively, wherein an input valueand a mapping value of the second pivot point are a maximum input valueand a maximum mapping value, respectively, and wherein input valuesbetween the minimum input value and the maximum input value are linearlymapped.
 4. The method of claim 3, wherein the image information includesinformation on the maximum input value and information on the maximummapping value, wherein the maximum input value is equal to a value ofthe information on the maximum input value, and wherein the maximummapping value is equal to a value of the information on the maximummapping value.
 5. The method of claim 3, wherein the information on thelinear mapping includes information on an input delta value of thesecond pivot point and information on a mapping delta value of thesecond pivot point, wherein the maximum input value is derived based onthe input delta value of the second pivot point, and wherein the maximummapping value is derived based on the mapping delta value of the secondpivot point.
 6. The method of claim 5, wherein the maximum mapping valueis derived based on the following equation,lmcs_max_mapped = (1<< bitdepthY) − I − lmcs_max_mapped_delta, herein,lmcs_max_mapped represents the maximum mapping value, bitdepthYrepresents a luma bit depth, and lmcs_max_mapped_delta represents themapping delta value of the second pivot point.
 7. The method of claim 1,wherein generating the mapped prediction luma samples comprises:deriving a forward mapping scaling factor for the prediction lumasamples; and generating the mapped prediction luma samples based on theforward mapping scaling factor, wherein the forward mapping scalingfactor is a single factor for the prediction luma samples.
 8. The methodof claim 7, wherein the forward mapping scaling factor is derived basedon the following equations,lmcsCWLinearAll = lmcs_max_mapped, Rounding = (lmcs_max_input_lmcs_min_input)>> 1, ScaleCoeffSingle = (lmcsCWLinearAll * (1<< FP_PREC) + Rounding))⁠/(lmcs_max_input − lmcs_min_input);,herein, lmcs_max_mapped represents the maximum mapping value,lmcs_min_input represents the minimum input value, lmcs_max_inputrepresents the maximum input value, ScaleCoeffSingle represents theforward mapping scaling factor, and FP_PREC represents a predeterminedconstant.
 9. The method of claim 8, wherein the mapped prediction lumasamples are generated based on the following equation,PredMapSamples[i][j] = (ScaleCoeffSingel * predSamples[i][j] + (1<< FP_PREC-1)))>> FP_PREC,herein, PredMapSamples[i][j] represents the mapped prediction lumasamples, ScaleCoeffSingle represents the forward mapping scaling factor,predSamples[i][j] represents the prediction luma samples, and FP_PREC isa predetermined constant.
 10. The method of claim 9, wherein the FP_PRECis equal to
 11. 11. The method of claim 1, wherein the method furthercomprises: deriving an inverse mapping scaling factor for thereconstructed luma samples; and generating modified reconstructed lumasamples based on the reconstructed luma samples and the inverse mappingscaling factor, wherein the inverse mapping scaling factor is a singlefactor for the reconstructed luma samples.
 12. The method of claim 11,wherein the inverse mapping scaling factor is derived based on thefollowing equations,lmcsCWLinearAll = lmcs_max_mapped, InvScaleCoeffSingle = (lmcs_max_input − lmcs_min_input) * (1<< FP_PREC)/lmcsCWLinearAll,herein, InvScaleCoeffSingle represents the inverse mapping scalingfactor, lmcs_min_input represents the input value of the first pivotpoint, lmcs_max_input represents the input value of the second pivotpoint, and FP_PREC represents a predetermined constant.
 13. The methodof claim 12, wherein the modified reconstructed luma samples aregenerated based on the following equation,invSample = lmcs_min_input + (InvScaleCoeffSingle * (lumaSample − lmcs_min_input) + (1<< (FP_PREC − 1)))>> FP_PREC,herein, invSample represents the modified reconstructed luma samples,lmcs_min_input represents the input value of the first pivot point,InvScaleCoeffSingle represents the inverse mapping scaling factor,lumaSample represents the reconstructed luma samples, and FP_PRECrepresents a predetermined constant.
 14. The method of claim 13, whereingenerating the scaled residual chroma samples comprises: deriving asingle chroma residual scaling factor; and generating the scaledresidual chroma samples based on the residual chroma samples and thesingle chroma residual scaling factor, wherein the single chromaresidual scaling factor is derived based on the following equation,ChromaScaleCoeffSingle = InvScaleCoeffSingle>> (FP_PREC − CSCALE_FP_PREC),herein, ChromaScaleCoeffSingle represents the single chroma residualscaling factor, InvScaleCoeffSingle represents the inverse mappingscaling factor, and FP_PREC and CSCALE_FP_PREC represent predeterminedconstants.
 15. The method of claim 14, wherein FP_PREC is equal to 11 or15, and wherein CSCALE_FP_PREC is equal to
 11. 16. The method of claim1, wherein the LMCS related information includes information on a numberof bins for deriving the mapped prediction luma samples, wherein anumber of pivot points for luma mapping is set equal to the number ofbins, wherein delta input values and delta mapping values of the pivotpoints by the number of bins are respectively signaled, wherein inputvalues and mapping values of the pivot points are derived based on thedelta input values and the delta mapping values, wherein the mappedprediction luma samples are generated based on the input values and themapping values.
 17. An image encoding method performed by an encodingapparatus, the method comprising: generating prediction luma samples andresidual chroma samples; generating mapped prediction luma samples;generating scaled residual chroma samples; deriving LMCS relatedinformation based on the mapped predicted luma samples and the scaledresidual chroma samples; generating residual luma samples based on themapped prediction luma samples; deriving residual information based onthe scaled residual chroma samples and the residual luma samples; andencoding image information including the LMCS related information andthe residual information, wherein the LMCS related information includesinformation on a linear LMCS, wherein the mapped prediction luma samplesare generated based on the information on the linear LMCS.
 18. Themethod of claim 17, wherein generating the scaled residual chromasamples includes: deriving a single chroma residual scaling factor; andgenerating the scaled residual chroma samples based on the residualchroma samples and the single chroma residual scaling factor; whereinthe image information includes information on the single chroma residualscaling factor, wherein a value of the information on the single chromaresidual scaling factor is equal to a value of the single chromaresidual scaling factor.
 19. The method of claim 17, wherein the LMCSrelated information includes information on a number of bins forderiving the mapped prediction luma samples, wherein a number of pivotpoints for luma mapping is set equal to the number of bins, whereindelta input values and delta mapping values of the pivot points aresignaled by the number of the bins, wherein input values and mappingvalues of the pivot points are derived based on the delta input valuesand the delta mapping values, wherein the mapped prediction luma samplesare generated based on the input values and the mapping values.
 20. Acomputer-readable digital storage medium, storing encoded informationcausing a decoding apparatus to perform an image decoding method, themethod comprising: obtaining image information including predictionrelated information, residual information, and luma mapping with chromascaling (LMCS) related information from a bitstream; generatingprediction luma samples based on the prediction related information;generating residual chroma samples based on the residual information;generating mapped prediction luma samples based on the LMCS relatedinformation and the prediction luma samples; generating reconstructedluma samples based on the mapped prediction luma samples; generatingscaled residual chroma samples based on the LMCS related information andthe residual chroma samples; and generating reconstructed chroma samplesbased on the scaled residual chroma samples, wherein the LMCS relatedinformation includes information on a linear LMCS, wherein the mappedprediction luma samples are generated based on the information on thelinear LMCS.